-- Logs begin at Tue 2022-07-19 10:18:56 UTC, end at Tue 2022-07-19 11:21:29 UTC. -- Jul 19 10:50:04.234747 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 19 10:50:04.242603 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Jul 19 10:50:04 2022] *** Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: nodename: dv-cinder-master-3-iscsi Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: machine: x86_64 Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: clock source: unix Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: pcre jit disabled Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: detected number of CPU cores: 4 Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: current working directory: / Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: detected binary path: /usr/bin/uwsgi-core Jul 19 10:50:04.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: your processes number limit is 47680 Jul 19 10:50:04.245300 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: your memory page size is 4096 bytes Jul 19 10:50:04.245300 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: detected max file descriptor number: 2048 Jul 19 10:50:04.245300 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: lock engine: pthread robust mutexes Jul 19 10:50:04.245300 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: thunder lock: enabled Jul 19 10:50:04.245300 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 19 10:50:04.245300 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Jul 19 10:50:04.288420 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: Python main interpreter initialized at 0x55bc65b059c0 Jul 19 10:50:04.288420 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: python threads support enabled Jul 19 10:50:04.288420 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: your server socket listen backlog is limited to 100 connections Jul 19 10:50:04.288420 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: your mercy for graceful operations on workers is 90 seconds Jul 19 10:50:04.288762 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: mapped 403077 bytes (393 KB) for 2 cores Jul 19 10:50:04.288858 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: *** Operational MODE: preforking *** Jul 19 10:50:04.289125 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: *** uWSGI is running in multiple interpreter mode *** Jul 19 10:50:04.289125 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: spawned uWSGI master process (pid: 80145) Jul 19 10:50:04.289308 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 19 10:50:04.292734 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: spawned uWSGI worker 1 (pid: 80152, cores: 1) Jul 19 10:50:04.292734 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: spawned uWSGI worker 2 (pid: 80153, cores: 1) Jul 19 10:50:04.292734 dv-cinder-master-3-iscsi devstack@keystone.service[80145]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 19 10:50:09.671683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 19 10:50:09.671683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: from cryptography.utils import int_from_bytes Jul 19 10:50:09.672469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 19 10:50:09.672469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: from cryptography.utils import int_from_bytes Jul 19 10:50:09.672919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 19 10:50:09.672919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: from cryptography.utils import int_from_bytes Jul 19 10:50:09.673020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 19 10:50:09.673020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: from cryptography.utils import int_from_bytes Jul 19 10:50:09.721260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 19 10:50:09.721417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 19 10:50:09.721494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 19 10:50:09.721574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 19 10:50:09.721651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] command line args: [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 19 10:50:09.721720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] command line args: [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 19 10:50:09.721895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 19 10:50:09.721981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 19 10:50:09.722057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ================================================================================ {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 19 10:50:09.722126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ================================================================================ {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 19 10:50:09.722314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] admin_token = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.722391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] admin_token = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.722557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] config_dir = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.722661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] config_dir = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.722738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.722885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.722964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] config_source = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] debug = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] config_source = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] debug = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.723939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_config_append = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.724953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_config_append = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_dir = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_file = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_dir = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_file = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.725975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.726950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.727946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.728985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.33/identity {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] publish_errors = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.33/identity {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.729951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.730073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] publish_errors = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.730153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.730283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.730938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] standard_threads = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.731946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] standard_threads = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] use_journal = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] use_json = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] use_stderr = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] use_journal = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] use_syslog = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] use_json = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.732989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] use_stderr = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.733254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] use_syslog = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.733328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.733484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 19 10:50:09.733560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.733738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.733814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.733974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.734936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.735974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.736991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.737866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.738876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.739941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740772 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.740949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.741895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.742883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.743937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.744956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.745893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.746071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.746700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.746814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.746916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.747932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.748974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.749912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.750933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.751925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.752942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753705 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.753953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.754962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.755913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.756873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.757993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.758884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.external = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.759907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.external = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.password = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.760878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] auth.token = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.password = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] auth.token = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.761922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.762920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.763959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.764979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.765972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.766416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.766559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.766683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.766805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.766927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.767920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.769919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.770930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.771886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.772917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.773977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.774962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.775964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.776972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.777975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.778985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.779891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.780926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.781916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.782909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.783909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.784919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.785916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.786943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.787934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.788955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.789955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.790897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.791017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.791643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.791785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.791914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.792851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.793457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.793583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.793708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.793844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.793980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.794990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.795884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.796009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.796433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] role.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.796560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.796682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] role.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.796821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.796941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] role.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] role.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.797970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.798999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.799882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.800924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.801958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.802956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.803959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.804962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.805914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.806940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.807932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.808079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.808202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.808329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.808454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.809962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.connection = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.810881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.connection = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.811916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.812959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.813995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.814887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80152) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 19 10:50:09.815662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 19 10:50:09.815662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80153) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 19 10:50:09.837623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.837853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.838055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.838190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.838322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.838498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.846318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.846715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.877936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.878191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.878445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:09.878675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.021174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.021738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.070274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.070507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.070660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.070898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.136207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.136430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.136658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.136807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.136952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.137854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.162316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.162487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.162629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.162714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.162898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.163935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.164066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.175238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.175395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.187972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.188895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.189147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.189279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.189410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.189410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.189652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80152) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.189800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80153) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 19 10:50:10.206496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.206704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 19 10:50:10.206833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.207008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 19 10:50:10.207147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.207274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 19 10:50:10.207397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.207522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.207646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 19 10:50:10.207769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.207907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.208049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.208181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.208310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.208474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.208474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.219768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.220875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.228627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.228877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.232468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.232757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.232888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.233929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.234935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.235017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.235194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.235274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.235451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.261561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.261955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.263185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.263463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.263555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.263817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.268716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.268999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.269079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.269274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.269368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.269612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.276526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.276832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.276913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.277120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.281938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.282222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.282302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.282499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.282593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.282771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.282851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.283904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.284085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.302912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.303182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.303263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.303491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.307741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.308021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.311483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.311682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.311765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.311971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.316800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.316993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.320373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.320557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.324168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.324357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.324465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.324650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.324772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.324958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.325050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.325241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.325323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.325509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.325589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.325779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.344952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.345134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.345319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.345504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.345598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.345768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.345849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.346027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.353701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.353920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.355409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.356164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.356258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.356434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.359264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.359490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.359572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.359764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.359845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.360037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.360128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.360313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.360396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.360592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.373728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.373972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.374057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.374248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.374329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.374521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.374611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.374873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.375063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.375143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.375343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.375425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.375632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.394374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.394738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.397676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.398045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.400970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.401347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.403078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.403463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.404717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.404993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.405112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.405273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.405384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.405659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.412732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.413043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.414334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.414653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.417570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.417857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.417939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.418676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.418759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.418957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.419051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.419249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.428140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.428422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.428504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.428711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.428794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.428992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.437062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80152) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.437341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.437433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80153) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 19 10:50:10.437618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.437698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.437889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.438952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439606 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.439995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.440082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.440246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.440325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.440500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.440580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80152) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.440784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80153) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 19 10:50:10.478697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55bc65b059c0 pid: 80152 (default app) Jul 19 10:50:10.478938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55bc65b059c0 pid: 80153 (default app) Jul 19 10:50:10.482510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf923a0-7e85-40f6-805f-a2a857f44987 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:10.482802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf923a0-7e85-40f6-805f-a2a857f44987 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:10.483078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf923a0-7e85-40f6-805f-a2a857f44987 None None] PATH_INFO: `/v3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:10.487239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1/1] 10.222.0.33 () {50 vars in 791 bytes} [Tue Jul 19 10:50:10 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:18.020011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:18.020568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:18.020568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:18.024839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1/2] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:18.029992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:18.030429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:18.030553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:18.099189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: INFO dbcounter [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] Registered counter for database keystone Jul 19 10:50:18.120878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG oslo_db.sqlalchemy.engines [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80152) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 19 10:50:18.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writer thread running {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 19 10:50:18.302702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG passlib.handlers.bcrypt [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80152) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 19 10:50:18.303045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG passlib.handlers.bcrypt [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80152) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 19 10:50:18.358072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:18.359691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:18.479852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 2/3] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 451 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 19 10:50:18.698503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:18.768969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: INFO dbcounter [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] Registered counter for database keystone Jul 19 10:50:18.788793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG oslo_db.sqlalchemy.engines [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80153) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 19 10:50:18.794978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writer thread running {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 19 10:50:18.912433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931dfa5b-4a32-4205-b473-f117384ef942 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:18.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:18.917428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:18.917709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:18.921051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:18.926816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:18.978353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING keystone.common.rbac_enforcer.enforcer [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 19 10:50:18.978634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00386277-14da-457d-b8f2-cd364ee30405 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:18.988514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 2/4] 10.222.0.33 () {58 vars in 1324 bytes} [Tue Jul 19 10:50:18 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 19 10:50:18.994420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:19.009265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2e060b-a71e-4eba-bea0-1b2a25abb8f3 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:19.014345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:19.015443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:19.015733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:19.016008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] PATH_INFO: `/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:19.020156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] RBAC: Authorizing `identity:get_project(project_id=a1dc1eb7c79345caaeb191fa0f9c008f)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:19.070580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING keystone.common.rbac_enforcer.enforcer [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 19 10:50:19.071104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab867dc-0b57-4c9e-81f5-fa3c4536ef9a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:19.075494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 3/5] 10.222.0.33 () {56 vars in 1207 bytes} [Tue Jul 19 10:50:18 2022] GET /identity/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:21.066142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1ee634-9d22-4d5e-a8bf-dd6eadb09bc0 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:21.066658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1ee634-9d22-4d5e-a8bf-dd6eadb09bc0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:21.066751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1ee634-9d22-4d5e-a8bf-dd6eadb09bc0 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:21.070733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 3/6] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:21.076030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:21.076999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:21.077157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:21.095607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:21.097177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:21.105007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 4/7] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:21 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 19 10:50:21.127179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1ee634-9d22-4d5e-a8bf-dd6eadb09bc0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:21.139456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1ee634-9d22-4d5e-a8bf-dd6eadb09bc0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:21.143571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd65e6b-f831-4a8c-aef9-65f262b90b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:21.144183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd65e6b-f831-4a8c-aef9-65f262b90b00 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:21.144465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd65e6b-f831-4a8c-aef9-65f262b90b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:21.144767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd65e6b-f831-4a8c-aef9-65f262b90b00 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:21.152271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd65e6b-f831-4a8c-aef9-65f262b90b00 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:21.154047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd65e6b-f831-4a8c-aef9-65f262b90b00 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:21.162614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 4/8] 10.222.0.33 () {58 vars in 1324 bytes} [Tue Jul 19 10:50:21 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 19 10:50:21.167721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:21.176529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d06b32-489e-4d73-b942-3144cf135a23 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:21.180886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976a77d8-9027-4799-942a-6a385dc535fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:21.181731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976a77d8-9027-4799-942a-6a385dc535fc admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:21.182019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976a77d8-9027-4799-942a-6a385dc535fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:21.182297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976a77d8-9027-4799-942a-6a385dc535fc admin admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:21.183689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976a77d8-9027-4799-942a-6a385dc535fc admin admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:21.186095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976a77d8-9027-4799-942a-6a385dc535fc admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:21.187582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 5/9] 10.222.0.33 () {56 vars in 1198 bytes} [Tue Jul 19 10:50:21 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:23.255372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de08e477-9f16-4cbc-b418-517c0e0443b9 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.255868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de08e477-9f16-4cbc-b418-517c0e0443b9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.255868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de08e477-9f16-4cbc-b418-517c0e0443b9 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.257326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 5/10] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:23.262489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.262873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.263050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.279493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4ddf8-1d52-4fa8-9eea-9ff60ffbed90 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.279781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4ddf8-1d52-4fa8-9eea-9ff60ffbed90 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.279988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:23.280078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4ddf8-1d52-4fa8-9eea-9ff60ffbed90 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.281220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:23.281400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 6/11] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:23.286507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.286797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.287077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.326816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG passlib.handlers.bcrypt [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80153) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 19 10:50:23.327232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG passlib.handlers.bcrypt [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80153) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 19 10:50:23.360997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 6/12] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:23.372351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:23.374480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:23.383925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.407571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:23.473440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 7/13] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:23.486772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa3a79c-04a3-4fc0-9b75-adde378a0d08 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:23.495359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.520882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:23.521604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.521887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.522162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.533606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:23.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:23.546050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:23.546846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 7/14] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:23.551907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.596344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c568c6a-5ea0-4806-b81c-a9d51a6fc77c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:23.598620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:23.633276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:23.635883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:23.636869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.637164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.637453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.651830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fed0885-6387-45e5-a7a7-d444ff9b7288 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:23.684470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:23.685152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.685435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.685737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.686484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:23.686789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:23.688167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:23.699770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 8/15] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:23.704009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:23.705566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:23.705975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.802129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 8/16] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 19 10:50:23.804428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea35b14-aaa3-47f3-bf17-83c0ea9e6ca0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:23.808298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.811014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:23.839283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:23.840140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.840424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.840728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.842546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:23.843984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:23.845640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 9/17] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:23.850957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.865292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:23.907121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15de8cf-585e-429b-9c25-e555640487b3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:23.945623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:23.946976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.947493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.947995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.950900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a754f33-b852-4fe8-9b6d-7bf700bbaa53 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:23.970819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:23.972295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:23.985571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:23.986358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:23.986645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:23.986918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:23.988094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:23.989587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:23.991213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 10/18] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:50:23.992495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:23.993358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:23.994308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:50:23.994308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:50:23.994308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:50:23.994308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:23.994308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 9/19] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:23.996895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:23.999647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:24.054390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:24.080627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:24.099195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e049599-e4ad-41d5-b63f-2cdabe93f7c5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:24.099719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810f30c4-028c-46a6-8228-993a9024109e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:24.132226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:24.132913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:24.133211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:24.133486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:24.133830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:24.134221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:24.134471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:24.134559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:24.134750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:24.135021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:24.135841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.domain.id=default, user.id=1eaf5dcf504f41ff99cb8d55073af0f9) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:24.135933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7401f6-f989-4c44-832b-5d7f6ca02652 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:24.136148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:24.137658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eb16b4-b10e-42d2-b1ef-923a6855982d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:24.145062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 11/20] 10.222.0.33 () {56 vars in 1322 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/role_assignments?user.id=1eaf5dcf504f41ff99cb8d55073af0f9&scope.domain.id=default&role.id=90dd7541fe26472fb3843454a738e154 => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:24.154532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 10/21] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:50:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:26.705520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57c8f4f-f3db-4f36-bf07-8a7affc5555c None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:26.706497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57c8f4f-f3db-4f36-bf07-8a7affc5555c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:26.706998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57c8f4f-f3db-4f36-bf07-8a7affc5555c None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:26.708841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 12/22] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:26.726514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170a073-6f09-4b9c-a35e-b5032e8a8994 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:26.727009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170a073-6f09-4b9c-a35e-b5032e8a8994 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:26.727086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170a073-6f09-4b9c-a35e-b5032e8a8994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:26.753950 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b170a073-6f09-4b9c-a35e-b5032e8a8994 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:26.755420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b170a073-6f09-4b9c-a35e-b5032e8a8994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:26.847394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 11/23] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:26.882799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57c8f4f-f3db-4f36-bf07-8a7affc5555c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:26.991048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cb8a02-6a2b-428e-80c7-c29b4b49996c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:26.991655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cb8a02-6a2b-428e-80c7-c29b4b49996c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:26.992128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cb8a02-6a2b-428e-80c7-c29b4b49996c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:26.993745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 12/24] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:27.008104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb10825c-0e89-4e23-9b0a-dd46535ea9b0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.008730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb10825c-0e89-4e23-9b0a-dd46535ea9b0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.009228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb10825c-0e89-4e23-9b0a-dd46535ea9b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.044204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bb10825c-0e89-4e23-9b0a-dd46535ea9b0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:27.046901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bb10825c-0e89-4e23-9b0a-dd46535ea9b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:27.074688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57c8f4f-f3db-4f36-bf07-8a7affc5555c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:27.117840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:27.118505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.118788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.119063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.119822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:27.120880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:27.122359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:27.152216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 13/25] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 19 10:50:27.153920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888c9472-900b-483c-a761-81ed59e99546 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.154492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888c9472-900b-483c-a761-81ed59e99546 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.154965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888c9472-900b-483c-a761-81ed59e99546 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.156417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 14/26] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:27.157789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f2a51-7172-49e1-acbb-a0b164912063 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.158318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f2a51-7172-49e1-acbb-a0b164912063 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.158802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f2a51-7172-49e1-acbb-a0b164912063 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.160462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 15/27] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:27.161877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d59d8-0f72-435f-b32d-4cd9a2fba845 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.162427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d59d8-0f72-435f-b32d-4cd9a2fba845 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.162910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d59d8-0f72-435f-b32d-4cd9a2fba845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.179739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b36d59d8-0f72-435f-b32d-4cd9a2fba845 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:27.181233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b36d59d8-0f72-435f-b32d-4cd9a2fba845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:27.272484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 16/28] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:27.278041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.278491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.278903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.282310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 13/29] 10.222.0.33 () {60 vars in 1131 bytes} [Tue Jul 19 10:50:26 2022] POST /identity/v3/roles => generated 219 bytes in 400 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:27.283490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:27.287897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:27.311304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:27.312529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:27.420683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f1c952-e27c-45d7-aa11-05eddd640ff9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:27.439589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 17/30] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:27.440473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:27.497920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:27.499176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.499463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.499742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.500512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:27.502201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:27.519746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:27.527236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 14/31] 10.222.0.33 () {60 vars in 1131 bytes} [Tue Jul 19 10:50:27 2022] POST /identity/v3/roles => generated 225 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:27.528947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:27.658979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600eed8c-2462-443b-9b1f-ccfe523e067b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:27.677965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:27.714365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:27.715505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.715979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.716442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.731358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:27.732199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1789e492-c868-49ba-8500-8034b3be629e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:27.734885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:50:27.763481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:27.764401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:27.765210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:50:27.765210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:50:27.765210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:50:27.765210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:27.765210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 18/32] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:27.771788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:27.772130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:27.772598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.772941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.773223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.787855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:27.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:27.797398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:27.798142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 15/33] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:27.803763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:27.891197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:27.911749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021240cb-3eab-4e36-986a-907659b74469 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:27.927547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07f2a9-d88f-489c-982a-3e360f406e06 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:27.951660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:27.952145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:27.952845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.953135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.953482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.954391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:27.954756 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:27.956136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873d442f-c1f4-41a7-9d2f-43df0ba1142e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:27.963306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:27.963966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.964252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.964538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.965330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:27.965642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:27.967742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f420f39d-e352-481a-83f2-dd215c62a1ee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:27.968211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 19/34] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:50:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:27.969502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f25f8ef-9526-474f-b861-b21305589326 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.969793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f25f8ef-9526-474f-b861-b21305589326 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.970077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f25f8ef-9526-474f-b861-b21305589326 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.971306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 20/35] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:27.972376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfb8aa-024e-4489-a2d6-3c141d726183 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.972667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfb8aa-024e-4489-a2d6-3c141d726183 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.973037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfb8aa-024e-4489-a2d6-3c141d726183 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.974265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 21/36] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:27 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:27.974943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfb8aa-024e-4489-a2d6-3c141d726183 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:27.978845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 16/37] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:27.980174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8026bb-4de0-4f42-8140-e2601d5e38b0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:27.980466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8026bb-4de0-4f42-8140-e2601d5e38b0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:27.980764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8026bb-4de0-4f42-8140-e2601d5e38b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:27.998280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-fe8026bb-4de0-4f42-8140-e2601d5e38b0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:27.999901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fe8026bb-4de0-4f42-8140-e2601d5e38b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:28.080767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 17/38] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:28.080767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.080767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.080767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.082039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfb8aa-024e-4489-a2d6-3c141d726183 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.084844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:28.086044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:28.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.118751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.119041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.119321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.120275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.121758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.146053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:28.152298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 18/39] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:28.153177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:28.200495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:28.253853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3610d-c426-4f3e-8b2f-3c0449d4e03d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.256093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 22/40] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:50:27 2022] POST /identity/v3/projects => generated 301 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:28.257074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:28.291182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.292079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.292367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.292646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.293979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.295499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.297394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 19/41] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:28 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:28.298006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:28.384808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ba29df-cf0e-4728-a082-0bb00909e88b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.413484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3e10b-e2cf-426f-be5a-f54155833fe8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.430083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.430254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.430462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.430724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.431479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.433081 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.433656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:28.448376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 23/42] 10.222.0.33 () {60 vars in 1131 bytes} [Tue Jul 19 10:50:28 2022] POST /identity/v3/roles => generated 223 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:28.449341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:28.461668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.462784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.462784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.464731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.464731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.465874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.467735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 20/43] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:28.473716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:28.523103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:28.567932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16eaa70-a208-4a4a-9b08-ee71d2079ca2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.595042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7d673-7fe7-471a-9952-b696f2b26393 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.604451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.605300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.605589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.605866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.607498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.609058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.610758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 24/44] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:28.617240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:28.735472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb9f714-f655-473a-9046-46ad223d2e5e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:28.737477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c42ebd-d3c2-43eb-a20b-2cd403009c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.738139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c42ebd-d3c2-43eb-a20b-2cd403009c37 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.738422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c42ebd-d3c2-43eb-a20b-2cd403009c37 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.738699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c42ebd-d3c2-43eb-a20b-2cd403009c37 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.739408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c42ebd-d3c2-43eb-a20b-2cd403009c37 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.740893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c42ebd-d3c2-43eb-a20b-2cd403009c37 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.762487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:28.769182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 21/45] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:50:28 2022] POST /identity/v3/projects => generated 312 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:50:28.771923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeb0ed7-2e46-4c32-be11-26a5cb8552c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:28.772919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeb0ed7-2e46-4c32-be11-26a5cb8552c1 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:28.773260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeb0ed7-2e46-4c32-be11-26a5cb8552c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:28.773580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeb0ed7-2e46-4c32-be11-26a5cb8552c1 None admin] PATH_INFO: `/v3/domains/default/users/1eaf5dcf504f41ff99cb8d55073af0f9/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:28.776059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfeb0ed7-2e46-4c32-be11-26a5cb8552c1 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=1eaf5dcf504f41ff99cb8d55073af0f9, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:28.777595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfeb0ed7-2e46-4c32-be11-26a5cb8552c1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:28.794715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 25/46] 10.222.0.33 () {58 vars in 1361 bytes} [Tue Jul 19 10:50:28 2022] PUT /identity/v3/domains/default/users/1eaf5dcf504f41ff99cb8d55073af0f9/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:30.799213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59f3029-d8eb-41d1-9a7e-9599b1920984 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:30.799727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59f3029-d8eb-41d1-9a7e-9599b1920984 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:30.799808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59f3029-d8eb-41d1-9a7e-9599b1920984 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:30.800282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571c6088-e3f6-4ab3-b246-3d47b06746e2 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:30.800656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571c6088-e3f6-4ab3-b246-3d47b06746e2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:30.800904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571c6088-e3f6-4ab3-b246-3d47b06746e2 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:30.801169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 22/47] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:30.802450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 26/48] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:30.805808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:30.806095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:30.806376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:30.807506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:30.807829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:30.808113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:30.822741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:30.823922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:30.824805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:30.826279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:30.850456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:30.891774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 23/49] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:30.892335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 27/50] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:30.915507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:30.915805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:31.017912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f6ec57-d0b6-4d1f-baac-7038bb181d67 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.020165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b839a31-80db-4b22-9483-5ff38624f698 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.034763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:31.052658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.053414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.053699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.053977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.054453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.055240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.055524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.055803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.057002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:31.058460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:31.060077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 24/51] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:31.064994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:31.065498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:31.073664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:31.074664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:31.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 28/52] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:31.080806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:31.100125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:31.170057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaf33c9-c3f2-41bb-9b81-dda6d0719ab7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.206917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.207597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.207885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.208162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.208937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:31.210411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:31.237713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 25/53] 10.222.0.33 () {60 vars in 1140 bytes} [Tue Jul 19 10:50:31 2022] POST /identity/v3/projects => generated 298 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:50:31.289015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9013c2b-33b5-4326-b209-5b81eebe7be0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.324669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.325328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.325589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.325870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.327140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:31.327449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:31.328900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:31.341994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 29/54] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:31.348305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:31.433135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:31.450192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9fa2b7-bf72-47db-8956-75d4f374f614 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.482963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.483830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.484111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.484382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.485602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:31.487035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:31.488649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 26/55] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:31 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:31.493915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:31.535090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:31.592723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285d4fb-ad62-4e07-a20a-e614fef7e689 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.630684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7964e259-5a70-4c4d-97d8-bcc350c0f3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.631576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7964e259-5a70-4c4d-97d8-bcc350c0f3ae None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.631927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7964e259-5a70-4c4d-97d8-bcc350c0f3ae None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.632287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7964e259-5a70-4c4d-97d8-bcc350c0f3ae None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.633568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7964e259-5a70-4c4d-97d8-bcc350c0f3ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:31.635028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7964e259-5a70-4c4d-97d8-bcc350c0f3ae None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:31.636661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 30/56] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:31.643079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:31.739773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea719d-2a6e-4b9e-85a8-7d7cdd23dfc0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:31.772990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:31.773634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:31.773919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:31.774196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:31.775025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.domain.id=default, user.id=1eaf5dcf504f41ff99cb8d55073af0f9) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:31.775337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:31.776794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd06dabf-1392-497b-9152-4a45f420cf88 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:31.782822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 27/57] 10.222.0.33 () {56 vars in 1322 bytes} [Tue Jul 19 10:50:31 2022] GET /identity/v3/role_assignments?user.id=1eaf5dcf504f41ff99cb8d55073af0f9&scope.domain.id=default&role.id=90dd7541fe26472fb3843454a738e154 => generated 530 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:33.279803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d48571a-64f3-4791-b5b5-07a3c62462b7 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:33.280282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d48571a-64f3-4791-b5b5-07a3c62462b7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:33.280282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d48571a-64f3-4791-b5b5-07a3c62462b7 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:33.281689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 31/58] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:33.286802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:33.287198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:33.287365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:33.305047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:33.306211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:33.317051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:33.370218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 28/59] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:33.395016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d48571a-64f3-4791-b5b5-07a3c62462b7 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:33.493485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d48571a-64f3-4791-b5b5-07a3c62462b7 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:33.502386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:33.526870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2fd09a-0e02-474c-b208-76266b54ec41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:33.527639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2fd09a-0e02-474c-b208-76266b54ec41 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:33.527926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2fd09a-0e02-474c-b208-76266b54ec41 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:33.528205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2fd09a-0e02-474c-b208-76266b54ec41 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:33.529385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2fd09a-0e02-474c-b208-76266b54ec41 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:33.531469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2fd09a-0e02-474c-b208-76266b54ec41 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:33.533021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 32/60] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:33.538264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:33.632449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eca2c28-7b9c-46bb-b910-75b883a5e688 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:33.665553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c26def-61e2-48a9-9833-29a801effb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:33.666200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c26def-61e2-48a9-9833-29a801effb70 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:33.666485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c26def-61e2-48a9-9833-29a801effb70 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:33.666758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c26def-61e2-48a9-9833-29a801effb70 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:33.667452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c26def-61e2-48a9-9833-29a801effb70 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:33.668906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c26def-61e2-48a9-9833-29a801effb70 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:33.692815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 29/61] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:50:33 2022] POST /identity/v3/users => generated 275 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:36.668116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b7b165-8e39-46b5-aa34-0ac894e96454 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:36.669136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b7b165-8e39-46b5-aa34-0ac894e96454 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:36.669644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b7b165-8e39-46b5-aa34-0ac894e96454 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:36.672048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 33/62] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:36.677254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f638b834-8e63-47db-b10d-d29eb183baa1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:36.678062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f638b834-8e63-47db-b10d-d29eb183baa1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:36.678505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f638b834-8e63-47db-b10d-d29eb183baa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:36.683828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:36.698299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f638b834-8e63-47db-b10d-d29eb183baa1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:36.699499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f638b834-8e63-47db-b10d-d29eb183baa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:36.783046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 30/63] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:36.822232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b7b165-8e39-46b5-aa34-0ac894e96454 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:36.969642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b7b165-8e39-46b5-aa34-0ac894e96454 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:36.972418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b7a81-75ae-4a23-87f6-734c10e111e3 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:36.972726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b7a81-75ae-4a23-87f6-734c10e111e3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:36.973015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b7a81-75ae-4a23-87f6-734c10e111e3 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:36.974392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 31/64] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:36.979319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a3807-bc6b-48dc-9357-fc7773724534 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:36.979609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a3807-bc6b-48dc-9357-fc7773724534 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:36.979891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a3807-bc6b-48dc-9357-fc7773724534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.007908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1b3a3807-bc6b-48dc-9357-fc7773724534 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:37.009125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1b3a3807-bc6b-48dc-9357-fc7773724534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:37.024264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.025083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.025367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.025637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.042237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:37.050490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:37.051306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:37.052042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:37.053333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 34/65] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:37.062565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.108544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.162948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 32/66] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:37.164550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4472d23f-5ddd-46ac-92e9-d0aadefd2249 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.165055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4472d23f-5ddd-46ac-92e9-d0aadefd2249 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.165554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4472d23f-5ddd-46ac-92e9-d0aadefd2249 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.166975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 33/67] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:37.173199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.173676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.174101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.198377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08e124-ae1b-4e8b-860c-a214d85a90ce None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.207794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:37.209241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:37.241615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.242885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.243450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.243863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.244922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:37.245384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:37.247134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:37.259128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 35/68] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:37.260279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.275035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.287281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 34/69] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:37.293663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.379550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.384894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d2cc66-eb24-4c2c-8511-03f4745e2206 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.399343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c9d3a5-ece4-460f-9a23-1d2fa68e3859 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.426462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca7feea-cec0-41f7-a4f9-830ccec5f386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.427260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca7feea-cec0-41f7-a4f9-830ccec5f386 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.427550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca7feea-cec0-41f7-a4f9-830ccec5f386 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.427831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca7feea-cec0-41f7-a4f9-830ccec5f386 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.436309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f6b04b-fe62-414a-a9f8-4a3b4a32f7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.437206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f6b04b-fe62-414a-a9f8-4a3b4a32f7ca None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.437492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f6b04b-fe62-414a-a9f8-4a3b4a32f7ca None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.437903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f6b04b-fe62-414a-a9f8-4a3b4a32f7ca None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.438357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca7feea-cec0-41f7-a4f9-830ccec5f386 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6ca7feea-cec0-41f7-a4f9-830ccec5f386 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:37.446061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:37.446886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 36/70] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:37.452859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3aa81-7247-41cc-9c01-353d795c302e None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.453552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3aa81-7247-41cc-9c01-353d795c302e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.453552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3aa81-7247-41cc-9c01-353d795c302e None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.453552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 37/71] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:37.453552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3aa81-7247-41cc-9c01-353d795c302e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.453766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f6b04b-fe62-414a-a9f8-4a3b4a32f7ca None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:37.454187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f6b04b-fe62-414a-a9f8-4a3b4a32f7ca None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:37.456149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 35/72] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:37.457562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.458032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.458481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.475565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:37.476796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:37.538929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.560510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3aa81-7247-41cc-9c01-353d795c302e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.561279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 36/73] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:37.562130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.614308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.615404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.615682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.625738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:37.633867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:37.634826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:37.635568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 38/74] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:37.636196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.655636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.662882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf743fc-3616-4f18-ba02-1bad2baf3a04 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.697204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.697865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.698163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.698421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.699171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:37.699477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:37.700877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada68aab-af62-4a84-82a3-017b822cc7fa None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:37.708925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 37/75] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:37.710424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfde171-9ec3-4680-a616-07e60a6ca3ff None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.710991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfde171-9ec3-4680-a616-07e60a6ca3ff None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.711088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfde171-9ec3-4680-a616-07e60a6ca3ff None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.712509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 38/76] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:37.713192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfde171-9ec3-4680-a616-07e60a6ca3ff None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.737676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829ebae9-6a13-4aec-8d98-62f9c86d2d7b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.758363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.770776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641fa653-36ba-4b61-af84-58a8de74d9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.771419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641fa653-36ba-4b61-af84-58a8de74d9d7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.771707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641fa653-36ba-4b61-af84-58a8de74d9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.771983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641fa653-36ba-4b61-af84-58a8de74d9d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.816782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfde171-9ec3-4680-a616-07e60a6ca3ff None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.839560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641fa653-36ba-4b61-af84-58a8de74d9d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:37.841071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641fa653-36ba-4b61-af84-58a8de74d9d7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:37.850029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:37.850818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.851100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.851378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.858843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:37.861587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:37.869306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:37.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:37.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 39/77] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:37.871645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:37.950452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 39/78] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:37.952265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:37.952554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:37.952858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:37.969419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:37.970514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:37.978685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa948a-e987-4658-b81e-24a60cc4e624 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:37.980213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.013942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.015131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.015420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.015698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.016448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:38.016786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.018150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.026079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 40/79] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:50:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.026870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.056720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 40/80] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:38.056720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.090402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.130071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c1c801-bf8e-4660-aeb0-0098735b9ac8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.140274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea16d92-d952-4d74-8e67-cb680ac5d94a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.163481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.164336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.164620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.164936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.166418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.167858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.169532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 41/81] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.170284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.173362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.174183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.174496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.174776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.175537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:38.175851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.177286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.185515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 41/82] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.186799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.206095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.274477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21024f14-cb88-41f3-b2ab-2b0d54b84fdd None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.284935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54306b1-9a6e-49bd-bad7-3d442b65cdf5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.294915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.307562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.308416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.308703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.309004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.310185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.312112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.315606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 42/83] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.316365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.324002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.324915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.325202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.325482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.327306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.328808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.330432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 42/84] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.331189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.411301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.414229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862df8ed-2b5f-49a3-bd18-9fe6cee8b4b4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.426861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987cb4f5-3b4f-46ce-8141-0b61dcb0943a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.447192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.447980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.448274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.448552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.449756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.451193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.453367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 43/85] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.457648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.464439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.465110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.465397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.465677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.510399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.532675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.534083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.551592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c5721-b9f0-4c4a-a94d-cf963547e84b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.583886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.584756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.585041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.585317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.586498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.587901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.589821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 44/86] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.594404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.624881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.633585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 43/87] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:38.634425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.689061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e37094-e71d-4c98-a1e3-57f775d70d2e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.714031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.722662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.723308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.723593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.723872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.724717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=89aa4c239d794594b3270fddfe326280) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:38.725036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.726458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.732354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 45/88] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/role_assignments?user.id=89aa4c239d794594b3270fddfe326280&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=168377268ae14ad882f881d40dd39b5c => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:38.733940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.736234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8984aff-3f2d-4082-998c-99d52d4b0a2c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.774214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.774982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.775168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.775448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.850064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.852992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.853166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7233d00-1e5d-4135-bb93-f94951f4af92 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:38.854599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.893887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:38.894682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:38.895661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:38.895661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:38.896079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:38.897630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:38.932734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 46/89] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:50:38 2022] POST /identity/v3/users => generated 289 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:38.932734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:38.954105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:38.987484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 44/90] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:38.988734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:39.074305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6562ec70-0c83-42e3-96b2-ee6b2f355cbf None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.124636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd6e02-6f3f-464c-8a7b-62ac24d261a9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.125455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.126377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.126670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.127784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.158032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:39.167109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.168404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.168591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.169983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.171284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.172821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.174736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 45/91] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:39.181125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:39.210628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.212137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.237031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:39.310875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4109560a-2847-438f-b99e-70f13f6087af None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.339841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 47/92] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:39.341252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:39.352641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.353840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.354301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.354748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.357036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.359495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.361444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 46/93] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:39 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:39.362454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:39.407608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:39.458110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9b1f0-6ad2-4d20-93eb-569519d497f6 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.494611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2546c6-ba3a-4a15-aec6-3932ff336aec None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.494897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.495544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.495833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.496109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.497212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=1eaf5dcf504f41ff99cb8d55073af0f9) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:39.497532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.499760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.505798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 48/94] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:39 2022] GET /identity/v3/role_assignments?user.id=1eaf5dcf504f41ff99cb8d55073af0f9&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:39.506596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:39.514584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:39.555480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.556371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.556653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.556957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] PATH_INFO: `/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.558280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] RBAC: Authorizing `identity:get_project(project_id=3660441d6e5e46a6a5a5a35e570c4c6b)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.560315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.566161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 47/95] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:39 2022] GET /identity/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b => generated 312 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:39.574111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:39.747193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb860e4-21a9-4fdd-88d5-2bc04151c594 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.785372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:39.803905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f0e12-360e-4b5e-88db-b6762d22b7b0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:39.818766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.819508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.819798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.820137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.821069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] RBAC: Adding query filter params (role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=89aa4c239d794594b3270fddfe326280) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:39.821386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.822841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31567873-123e-4b94-a1c4-579dab946a7b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.837673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 49/96] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:39 2022] GET /identity/v3/role_assignments?user.id=89aa4c239d794594b3270fddfe326280&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 259 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:39.883658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:39.884348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:39.884632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:39.884936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:39.885811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=3660441d6e5e46a6a5a5a35e570c4c6b, user.id=89aa4c239d794594b3270fddfe326280) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:39.886126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:39.887627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfc3d48-7339-46bc-928c-0ff6f92e81ea None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:39.899294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 48/97] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:39 2022] GET /identity/v3/role_assignments?user.id=89aa4c239d794594b3270fddfe326280&scope.project.id=3660441d6e5e46a6a5a5a35e570c4c6b&role.id=168377268ae14ad882f881d40dd39b5c => generated 259 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:40.876426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19144f97-da98-4991-9424-65106cee0dc7 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:40.876957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19144f97-da98-4991-9424-65106cee0dc7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:40.877062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19144f97-da98-4991-9424-65106cee0dc7 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:40.878524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 50/98] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:40.883857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:40.884723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:40.885221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:40.902182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:40.903668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:40.962857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:40.994013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 49/99] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:41.028512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19144f97-da98-4991-9424-65106cee0dc7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:41.170369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19144f97-da98-4991-9424-65106cee0dc7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:41.221355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:41.222166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:41.222456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:41.222736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:41.241506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:41.249253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:41.250087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:41.251258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 51/100] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:41.256246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:41.401133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234890e5-1db1-417c-8fad-3d28bc52ad64 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:41.436418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:41.437121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:41.437402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:41.437680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:41.438431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:41.438737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:41.440134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:41.448364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 50/101] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:41.453919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:41.522983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:41.604938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8660c8-bd15-4b92-8a59-c88854e93b8b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:41.654322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:41.655765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:41.656056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:41.656336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:41.657586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:41.659056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:41.661021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 52/102] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:41 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:41.666160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:41.753178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:41.811680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dcfb32-be01-40f4-b08b-01f213804da1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:41.847113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2db7d31-ddf4-4344-bab6-1dc85910f9fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:41.847983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db7d31-ddf4-4344-bab6-1dc85910f9fc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:41.848268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db7d31-ddf4-4344-bab6-1dc85910f9fc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:41.848543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db7d31-ddf4-4344-bab6-1dc85910f9fc None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:41.849760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2db7d31-ddf4-4344-bab6-1dc85910f9fc None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:41.851263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2db7d31-ddf4-4344-bab6-1dc85910f9fc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:41.853257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 51/103] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:41 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:41.859397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:41.994072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bd66e6-4898-4aaf-8966-b0cece936e5c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:41.995032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bd66e6-4898-4aaf-8966-b0cece936e5c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:41.995514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bd66e6-4898-4aaf-8966-b0cece936e5c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:41.997124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 52/104] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:42.002885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a7f5e8-e140-4670-a77f-9e2fc2bdbb8f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.003514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a7f5e8-e140-4670-a77f-9e2fc2bdbb8f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.003984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a7f5e8-e140-4670-a77f-9e2fc2bdbb8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.021593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-00a7f5e8-e140-4670-a77f-9e2fc2bdbb8f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:42.023110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-00a7f5e8-e140-4670-a77f-9e2fc2bdbb8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:42.042864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b417e6-5f42-4bb8-aec9-2fad812eff41 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:42.072029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:42.104827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:42.106791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.107319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.107510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 53/105] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:42.108117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/89aa4c239d794594b3270fddfe326280/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.109148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38726c2-d8bd-4713-8f1d-9fcfd69732f4 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.109439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38726c2-d8bd-4713-8f1d-9fcfd69732f4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.109718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38726c2-d8bd-4713-8f1d-9fcfd69732f4 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.111029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] RBAC: Authorizing `identity:create_grant(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user_id=89aa4c239d794594b3270fddfe326280, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:42.111130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 54/106] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:42.116594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:42.117862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.118151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.118481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.145643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 53/107] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:41 2022] PUT /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/89aa4c239d794594b3270fddfe326280/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:42.154113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:42.155384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:42.166246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:42.189264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:42.272822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 55/108] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:42.342488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042368d2-05db-4dbe-a3ea-8825e91c2fee None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:42.364572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:42.390548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:42.391370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.391659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.391942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.393237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:42.394878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:42.397118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 54/109] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:42.411183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:42.486084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:42.526578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e12bca6-4b3e-444c-a7dd-367981b8c6ac None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:42.580260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:42.581448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.582618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.583090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.603600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:42.614064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:42.614933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:42.615695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:42.625097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79def891-b7f0-4dfb-b47a-e3eff1c4a332 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:42.626019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 56/110] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:42 2022] GET /identity/v3/roles/reader => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:42.632751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:42.677775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:42.684836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:42.685784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.686359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.687220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.688553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:42.690278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:42.741828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 55/111] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:50:42 2022] POST /identity/v3/projects => generated 302 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:42.780830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dd3ffe-051a-4e00-8218-105152391b90 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:42.826219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:42.827057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.827420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.827705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.828525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:42.828920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:42.830395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b606bd8-eae3-439f-9499-a1f0655253a8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:42.839817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 57/112] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:42.846587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:42.927696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e77441-4186-49c8-975c-71bf04004f7d None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.928330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e77441-4186-49c8-975c-71bf04004f7d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.928856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e77441-4186-49c8-975c-71bf04004f7d None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.930449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 58/113] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:42.942241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:42.942779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:42.943249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:42.981897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:42.984322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:43.025750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.050644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef2dc5d-1cff-4189-9b25-68b41b1c2b9a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.097230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 59/114] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:43.098469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.099366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.099650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.099928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] PATH_INFO: `/v3/users/0fcaa119b56d42a19ed02a44c602c5c1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.115222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] RBAC: Authorizing `identity:get_user(user_id=0fcaa119b56d42a19ed02a44c602c5c1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:43.116772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:43.118492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 56/115] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:42 2022] GET /identity/v3/users/0fcaa119b56d42a19ed02a44c602c5c1 => generated 289 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:43.124029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.141640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.165091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.260010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94bbb2-ef9f-4300-bb71-d45f543a85b5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.283767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f263b4-7bee-4de5-8e38-938d86dfcd28 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.298015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443807b4-7ebe-463f-a14e-00cd1eae4ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.298831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443807b4-7ebe-463f-a14e-00cd1eae4ee4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.299121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443807b4-7ebe-463f-a14e-00cd1eae4ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.299397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443807b4-7ebe-463f-a14e-00cd1eae4ee4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.310410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443807b4-7ebe-463f-a14e-00cd1eae4ee4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-443807b4-7ebe-463f-a14e-00cd1eae4ee4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:43.319408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:43.320231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:43.321068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 57/116] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:43.322328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750c104e-d82b-4f5f-820c-b91f9eeeaa4f None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.322624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750c104e-d82b-4f5f-820c-b91f9eeeaa4f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.322904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750c104e-d82b-4f5f-820c-b91f9eeeaa4f None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.324181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 58/117] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:43.327144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750c104e-d82b-4f5f-820c-b91f9eeeaa4f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.330755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b774e7a-132c-418e-b4d5-4874929b56cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.331647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b774e7a-132c-418e-b4d5-4874929b56cb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.332160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b774e7a-132c-418e-b4d5-4874929b56cb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.332628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b774e7a-132c-418e-b4d5-4874929b56cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.341273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.406645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b774e7a-132c-418e-b4d5-4874929b56cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:43.408442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b774e7a-132c-418e-b4d5-4874929b56cb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:43.427641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.438138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750c104e-d82b-4f5f-820c-b91f9eeeaa4f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.473613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.474325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.474618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.474927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.475638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:43.475946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:43.477364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d1314b-91d5-446e-b576-72ed7e72a320 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:43.486853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 59/118] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:43.488105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.488449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.488751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.506313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:43.507464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:43.518933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 60/119] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:43.520538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d77b8-357e-479f-9fdf-e0a2c1c2ae7c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.521062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d77b8-357e-479f-9fdf-e0a2c1c2ae7c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.521541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d77b8-357e-479f-9fdf-e0a2c1c2ae7c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.523017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 61/120] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:43.523889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7d77b8-357e-479f-9fdf-e0a2c1c2ae7c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.582047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 60/121] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:43.582824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.585029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.642610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7d77b8-357e-479f-9fdf-e0a2c1c2ae7c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.662178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.682092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d41437-3418-4442-a503-e31112dc27d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.683074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d41437-3418-4442-a503-e31112dc27d8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.683371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d41437-3418-4442-a503-e31112dc27d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.683652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d41437-3418-4442-a503-e31112dc27d8 None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.685009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d41437-3418-4442-a503-e31112dc27d8 None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:43.686708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d41437-3418-4442-a503-e31112dc27d8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:43.688537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76eaf2b7-03d2-4e89-b2ce-78b0d2a9a9a9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.688667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 62/122] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:43.689841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.690245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.690539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.710027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:43.712037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:43.728280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.729259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.729612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.730024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.731371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:43.733893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:43.735753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 61/123] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:43.741695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.788669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 63/124] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:43.797149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.828214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.876774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2444a7-bb75-4988-8f63-44589064c425 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.923487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8684593-efe2-4119-b80b-3dee30eb1844 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:43.935940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.937399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.937695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.937977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.946653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:43.949677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:43.957496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:43.958419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:43.959169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 62/125] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:43.960444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:43.972544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:43.974211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:43.974969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:43.975456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:43.977684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:43.979949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:43.982360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 64/126] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:43.984091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.080851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068f330-f159-4e35-8a70-6ab6502b0b14 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.099424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96067f9d-78d6-42ec-a561-c10ea58c08bd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.119126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:44.128191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.129512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.130002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.130689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.131846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] RBAC: Adding query filter params (role.id=0dd145a4768f4049b471a6b9c1ce4562, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=0fcaa119b56d42a19ed02a44c602c5c1) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:44.132352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.134683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.137400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.138260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.138883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.138883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.142302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 63/127] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/role_assignments?user.id=0fcaa119b56d42a19ed02a44c602c5c1&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=0dd145a4768f4049b471a6b9c1ce4562 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:44.143257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.165747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:44.181221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:44.182150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:44.182898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 65/128] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:44.183520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.265663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a87162-fab5-4fb5-b4f3-bd955e4a45ee None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.287064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:44.307929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.308616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.308923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.309210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.309979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:44.310376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.311852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.321427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 64/129] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:44.322326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.391558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:44.406019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256ff3cd-cf1e-405f-9db7-c898fca54c97 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.454838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed18a12-5ab6-4916-a6b5-b7fa5495edcb None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.478786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.479694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.479980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.480256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/89aa4c239d794594b3270fddfe326280/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.486585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] RBAC: Authorizing `identity:create_grant(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user_id=89aa4c239d794594b3270fddfe326280, role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.488057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.496122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.496875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.497168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.497468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.498232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] RBAC: Adding query filter params (name=member) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:44.498543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.500452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.512966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 66/130] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:44 2022] PUT /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/89aa4c239d794594b3270fddfe326280/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:44.513946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.517685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 65/131] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:44.523656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.627865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c8b118-fc16-455c-88d9-6d55dd53274a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.630485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:44.667582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.669136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.669424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.669703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.672289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.673793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.675470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 67/132] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:44 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:44.681408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.740720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18954df1-cd77-4241-8e48-e0e763505bb8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.766225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:44.794871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd8eef2-1a51-410b-8a29-7ef448a501d1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.801978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.803167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.803664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.804145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.809532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.811857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.814286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 66/133] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:44 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:44.819889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.833192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.834119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.834407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.834742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.835989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.837485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:44.839648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 68/134] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:44 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:44.848721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:44.891806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:44.951108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee2804b-1b22-4bd1-96ff-a7f83993387c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.956363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffc22e-e9fa-478a-8916-b228e2f72176 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:44.992926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0490d9e-d013-479f-b771-e4b23dcd1369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:44.993874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0490d9e-d013-479f-b771-e4b23dcd1369 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:44.994158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0490d9e-d013-479f-b771-e4b23dcd1369 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:44.994437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0490d9e-d013-479f-b771-e4b23dcd1369 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/1eaf5dcf504f41ff99cb8d55073af0f9/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:44.996442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0490d9e-d013-479f-b771-e4b23dcd1369 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user_id=1eaf5dcf504f41ff99cb8d55073af0f9, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:44.997940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0490d9e-d013-479f-b771-e4b23dcd1369 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:45.001450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:45.002571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.003081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.003558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] PATH_INFO: `/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.005062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] RBAC: Authorizing `identity:get_project(project_id=3660441d6e5e46a6a5a5a35e570c4c6b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:45.006692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:45.008605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 67/135] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:44 2022] GET /identity/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b => generated 312 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:45.019614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 69/136] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:44 2022] PUT /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/1eaf5dcf504f41ff99cb8d55073af0f9/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 10:50:45.021780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:45.058492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f752303b-4c3f-467a-adfe-8d5b875f35ba None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.058862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f752303b-4c3f-467a-adfe-8d5b875f35ba None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.059217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f752303b-4c3f-467a-adfe-8d5b875f35ba None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.060995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 70/137] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:45.073496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.073877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.074231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.108914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:45.109422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:45.109422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:45.212537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 71/138] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:45.237336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:45.238645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28877483-d115-4dac-9c7f-d304adfb6e84 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:45.269810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:45.325464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:45.326378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.326666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.327480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] PATH_INFO: `/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b/users/89aa4c239d794594b3270fddfe326280/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.333276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] RBAC: Authorizing `identity:create_grant(project_id=3660441d6e5e46a6a5a5a35e570c4c6b, user_id=89aa4c239d794594b3270fddfe326280, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:45.334736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:45.346457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4075fa17-caae-4382-92e0-16e484a5367d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:45.380045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 68/139] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:45 2022] PUT /identity/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b/users/89aa4c239d794594b3270fddfe326280/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:45.387329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bfdcc6-afe8-4b96-81bb-ca583a276820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:45.388195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfdcc6-afe8-4b96-81bb-ca583a276820 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.388490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfdcc6-afe8-4b96-81bb-ca583a276820 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.388794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfdcc6-afe8-4b96-81bb-ca583a276820 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.406407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bfdcc6-afe8-4b96-81bb-ca583a276820 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-74bfdcc6-afe8-4b96-81bb-ca583a276820 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:45.418142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:45.419030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:45.419798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:45.420973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 72/140] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:45.426135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:45.566241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123f59d-408c-4ebd-891f-68d98c966d6c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123f59d-408c-4ebd-891f-68d98c966d6c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.566737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123f59d-408c-4ebd-891f-68d98c966d6c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.569174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 73/141] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:45.573517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.573810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.574145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.610228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:45.611443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:45.621149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea2968-31d1-4fc7-b390-46afda481d37 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:45.653311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:45.704503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:45.706408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.706408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.706408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:45.706609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] RBAC: Adding query filter params (name=member) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:45.716290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:45.716290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:45.731088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 69/142] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:45.737977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:45.780424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 74/143] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:45.813243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:45.849160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:45.916021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2953057e-c5b3-443b-af3d-53341ee442b4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:45.979413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bd597d-41c3-4552-82d8-56dcb119f106 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:45.993950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:45.995166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:45.995634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:45.996086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:46.001472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:46.003199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:46.005377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 70/144] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:45 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:46.010909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:46.043083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:46.044140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:46.044658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:46.045103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:46.048398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:46.050043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:46.052620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 75/145] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:46.057984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:46.133132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:46.261761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3619af-d9ac-4123-8793-0bb79a8a40f1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:46.269039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:46.283190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c25081b-5242-47be-a033-51141b991f57 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:46.354219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:46.354913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:46.355195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:46.355467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:46.356449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:46.360186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:46.363468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:46.364116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:46.364397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:46.364671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:46.425746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 76/146] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:50:46 2022] POST /identity/v3/users => generated 283 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:46.485257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:46.486787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:46.557058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:46.624013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 71/147] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 613 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:46.629934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:46.768882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9369c3-46bf-4213-812e-12f789d7e042 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:46.806779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c84e23-d89a-4df7-a390-6f8e772428ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:46.807636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c84e23-d89a-4df7-a390-6f8e772428ea None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:46.807915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c84e23-d89a-4df7-a390-6f8e772428ea None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:46.808186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c84e23-d89a-4df7-a390-6f8e772428ea None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:46.813432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c84e23-d89a-4df7-a390-6f8e772428ea None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:46.814908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c84e23-d89a-4df7-a390-6f8e772428ea None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:46.816612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 77/148] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:46 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:46.822421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:46.955862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957de7ba-7488-45bf-b798-c08d76ec2e39 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:47.002687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:47.003344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.003627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.003902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.004759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=89aa4c239d794594b3270fddfe326280) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:47.005070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:47.006567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea5ed4-ab81-48c1-9638-6f672a499681 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:47.015175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 72/149] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:46 2022] GET /identity/v3/role_assignments?user.id=89aa4c239d794594b3270fddfe326280&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=168377268ae14ad882f881d40dd39b5c => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:47.120535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f83fc4-8ee1-4f8b-aeae-ca44c7708fd0 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.120887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f83fc4-8ee1-4f8b-aeae-ca44c7708fd0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.121079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f83fc4-8ee1-4f8b-aeae-ca44c7708fd0 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.122656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 78/150] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:47.128657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.129227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.129689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.153867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:47.156354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:47.158929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:47.231728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 73/151] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:47.271927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f83fc4-8ee1-4f8b-aeae-ca44c7708fd0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:47.288021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:47.393121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f83fc4-8ee1-4f8b-aeae-ca44c7708fd0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:47.430445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:47.431248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.431536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.431811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.443306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:47.451166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:47.452643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:47.453459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 79/152] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:47.459014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:47.561861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0466c5-e3c2-43eb-8ca9-57957616385f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:47.597791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:47.598472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.598754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.599030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.599787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:47.600092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:47.601496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:47.609061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:47.615625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 74/153] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:47.617026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:47.690018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:47.721969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a43da0-b255-491a-ad7d-03c06a84af0d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:47.756317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:47.757225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.757506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.757780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] PATH_INFO: `/v3/users/0fcaa119b56d42a19ed02a44c602c5c1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.758980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] RBAC: Authorizing `identity:get_user(user_id=0fcaa119b56d42a19ed02a44c602c5c1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:47.760466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:47.762146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 80/154] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:47 2022] GET /identity/v3/users/0fcaa119b56d42a19ed02a44c602c5c1 => generated 289 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:47.768281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:47.873424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799015a-e184-4e5d-b76d-96a93dc36687 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:47.909255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb34ee89-b20f-4681-be0b-c8020e0a065a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:47.910127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb34ee89-b20f-4681-be0b-c8020e0a065a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:47.910414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb34ee89-b20f-4681-be0b-c8020e0a065a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:47.910690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb34ee89-b20f-4681-be0b-c8020e0a065a None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:47.911946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb34ee89-b20f-4681-be0b-c8020e0a065a None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:47.913458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb34ee89-b20f-4681-be0b-c8020e0a065a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:47.916325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 75/155] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:47 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:47.924896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:48.030203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7206b3-554a-4391-8854-6c7a81ef722d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:48.064872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ba55e3-565c-4d18-8ae4-0fdb18a7ec3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:48.065830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ba55e3-565c-4d18-8ae4-0fdb18a7ec3f None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.066171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ba55e3-565c-4d18-8ae4-0fdb18a7ec3f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.066444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ba55e3-565c-4d18-8ae4-0fdb18a7ec3f None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/0fcaa119b56d42a19ed02a44c602c5c1/roles/0dd145a4768f4049b471a6b9c1ce4562` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.068502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ba55e3-565c-4d18-8ae4-0fdb18a7ec3f None admin] RBAC: Authorizing `identity:create_grant(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user_id=0fcaa119b56d42a19ed02a44c602c5c1, role_id=0dd145a4768f4049b471a6b9c1ce4562)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:48.069989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ba55e3-565c-4d18-8ae4-0fdb18a7ec3f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:48.078963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a3b012-651c-42d3-8db1-99d2e31a250f None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.079258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a3b012-651c-42d3-8db1-99d2e31a250f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.079512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a3b012-651c-42d3-8db1-99d2e31a250f None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 76/156] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:48.086661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:48.086828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.086955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.087879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.090962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 81/157] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:47 2022] PUT /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/users/0fcaa119b56d42a19ed02a44c602c5c1/roles/0dd145a4768f4049b471a6b9c1ce4562 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:48.113615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:48.115174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:48.135075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ece3c-29c7-484b-86f7-bcda3104a494 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.135626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ece3c-29c7-484b-86f7-bcda3104a494 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.136063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ece3c-29c7-484b-86f7-bcda3104a494 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.138008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 82/158] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:48.146519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.147086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.147545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.201127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:48.206077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:48.206799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 77/159] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:48.207398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:48.274047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:48.386402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 83/160] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:48.449054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:48.492149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106252f3-7bed-4bad-acb7-cdf074b3c2d1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:48.562606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:48.563415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.563699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.563977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.593945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:48.606253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:48.607137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:48.607901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:48.618358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713c5e3-c383-4112-b4dd-79258158f0e7 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:48.619565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 78/161] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:48.637915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:48.670457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:48.704721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:48.704721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.704721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.704721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.740644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:48.744337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:48.753976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:48.754925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:48.755738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:48.758899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 84/162] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:50:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:48.790151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:48.806818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e48dec3-8828-4b62-a4f4-7bd5f5c97229 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:48.868638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:48.869710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:48.869710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:48.869878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:48.872721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:48.872721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:48.872721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:48.907060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 79/163] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:48.917566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:49.045375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4708e17f-6d03-4be3-bb3b-075adb143be1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:49.120270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:49.121009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.121295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.121568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.122433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:49.122742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:49.124816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1194f-75d2-4686-804c-d3daa3e4f26f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:49.139935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 85/164] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:50:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:49.141466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e334b6-48e7-40e2-b545-10a491a36d64 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.141773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e334b6-48e7-40e2-b545-10a491a36d64 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.142006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e334b6-48e7-40e2-b545-10a491a36d64 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.143233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 86/165] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:49.148658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.149281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.149563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.155134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcc3bd0-a5bd-4978-8e2a-844ed44656d6 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:49.180063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:49.181651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:49.192711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:49.235228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:49.236870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.237373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.237836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] PATH_INFO: `/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.245409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] RBAC: Authorizing `identity:get_user(user_id=1eaf5dcf504f41ff99cb8d55073af0f9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:49.247071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:49.249490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 80/166] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:48 2022] GET /identity/v3/users/1eaf5dcf504f41ff99cb8d55073af0f9 => generated 247 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:49.250456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:49.285899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:49.342595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 87/167] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:49.343722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:49.504897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15890ee-1204-446f-8b6b-ce6cd41d8c88 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:49.569416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:49.570607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.570767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f072c8-13b9-4dbe-beb3-2ac4adc636e0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:49.571578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.571995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.573686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:49.575281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:49.578936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 81/168] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:49 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:49.579988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:49.653424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:49.654182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.654544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.654878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.668219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:49.737878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e3f188-63e5-44d2-9bce-c32012579ace None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:49.773517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:49.798853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:49.800004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:49.800432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:49.800968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:49.801560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:49.803222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:49.817014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:49.825972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:49.827049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:49.827996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:49.830812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 82/169] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:49.832035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:49.989831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 88/170] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 646 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:49.990726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:50.006270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc0d33-b96b-4a95-999a-2ad829efe919 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.036425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:50.070057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdae50f-c69c-4192-9ab2-03db44b758ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.070718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdae50f-c69c-4192-9ab2-03db44b758ae None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.070998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdae50f-c69c-4192-9ab2-03db44b758ae None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.071271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdae50f-c69c-4192-9ab2-03db44b758ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.161613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:50.173928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdae50f-c69c-4192-9ab2-03db44b758ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:50.175630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdae50f-c69c-4192-9ab2-03db44b758ae None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:50.201413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5368b0b2-07a1-416b-af86-17cf2cac0e06 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.267493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.268165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.268447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.268747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.269503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:50.269813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:50.271413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:50.286977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 89/171] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:50.287733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:50.288752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 83/172] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:50.290936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd304f7-497c-47ec-8ebc-b55cb4582db7 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.292262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd304f7-497c-47ec-8ebc-b55cb4582db7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.292796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd304f7-497c-47ec-8ebc-b55cb4582db7 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.294508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 84/173] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:50.295393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd304f7-497c-47ec-8ebc-b55cb4582db7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:50.319247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:50.464464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c06b0f-f3b1-4f2f-9c43-39c8cc8ca840 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.468718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd304f7-497c-47ec-8ebc-b55cb4582db7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.530048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.530936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.531214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.531482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.534442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:50.535903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:50.537697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 90/174] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:50 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:50.538397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:50.540281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a1fe45-a06c-41f8-a7e0-cf733c668261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.541181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1fe45-a06c-41f8-a7e0-cf733c668261 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.541541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1fe45-a06c-41f8-a7e0-cf733c668261 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.541778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1fe45-a06c-41f8-a7e0-cf733c668261 None admin] PATH_INFO: `/v3/users/89aa4c239d794594b3270fddfe326280` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.543190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a1fe45-a06c-41f8-a7e0-cf733c668261 None admin] RBAC: Authorizing `identity:get_user(user_id=89aa4c239d794594b3270fddfe326280)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:50.544676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a1fe45-a06c-41f8-a7e0-cf733c668261 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:50.548759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 85/175] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:50 2022] GET /identity/v3/users/89aa4c239d794594b3270fddfe326280 => generated 275 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:50.550864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.551143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.551413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.575285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:50.593701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:50.595213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:50.668279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc768251-fa28-4324-991b-f786cc8298b4 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.707921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.708843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.709127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.709401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.710669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:50.712241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:50.714285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 91/176] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:50 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:50.714960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:50.722302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 86/177] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:50.723744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:50.748202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:50.831594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:50.872536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491c9c3-bcef-45d2-bca4-b1393c16d09e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.881971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338e324-12ab-4688-80ac-33db2e55ba2d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:50.948602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.949917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.950673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.951928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cb7a43-ca69-4462-bd77-46d4fa915409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:50.953028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.953817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb7a43-ca69-4462-bd77-46d4fa915409 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:50.954801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=1eaf5dcf504f41ff99cb8d55073af0f9) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:50.955236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb7a43-ca69-4462-bd77-46d4fa915409 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:50.955682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:50.956089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb7a43-ca69-4462-bd77-46d4fa915409 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:50.958259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:50.972303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 92/178] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:50 2022] GET /identity/v3/role_assignments?user.id=1eaf5dcf504f41ff99cb8d55073af0f9&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:50.975098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.101316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb7a43-ca69-4462-bd77-46d4fa915409 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:51.103785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb7a43-ca69-4462-bd77-46d4fa915409 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:51.142034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.164457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d51f7a9-6c56-4f60-b473-465d6752dc5d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.204948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.205858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.206280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.206688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.207795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] RBAC: Adding query filter params (role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=89aa4c239d794594b3270fddfe326280) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:51.208263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:51.209729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:51.216014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 93/179] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:50 2022] GET /identity/v3/role_assignments?user.id=89aa4c239d794594b3270fddfe326280&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 608 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:51.216639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.228849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.250189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 87/180] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:51.251762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742913c-0a78-4dc9-b8a0-d34eac4928d9 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.252259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742913c-0a78-4dc9-b8a0-d34eac4928d9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.252812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742913c-0a78-4dc9-b8a0-d34eac4928d9 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.254299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 88/181] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:51.255689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3815d09-d749-4648-8681-cb6401f46c65 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.256129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3815d09-d749-4648-8681-cb6401f46c65 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.256554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3815d09-d749-4648-8681-cb6401f46c65 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.258103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 89/182] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:51.258984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3815d09-d749-4648-8681-cb6401f46c65 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.362190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7081fb22-2de7-4a4d-938b-096377ddeaa4 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.394173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3815d09-d749-4648-8681-cb6401f46c65 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.405575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac35ee7-7695-4e18-b504-1a420a70135f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.406417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac35ee7-7695-4e18-b504-1a420a70135f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.406702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac35ee7-7695-4e18-b504-1a420a70135f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.406978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac35ee7-7695-4e18-b504-1a420a70135f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.414151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.420416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac35ee7-7695-4e18-b504-1a420a70135f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-eac35ee7-7695-4e18-b504-1a420a70135f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:51.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:51.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:51.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 94/183] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:51.431556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b59ce-5fb5-431b-8519-ac8a98c18322 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.431868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b59ce-5fb5-431b-8519-ac8a98c18322 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.432145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b59ce-5fb5-431b-8519-ac8a98c18322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.441904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e126d73b-57c2-4db6-a74e-056fc325fcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.442772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e126d73b-57c2-4db6-a74e-056fc325fcaf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.443052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e126d73b-57c2-4db6-a74e-056fc325fcaf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.443324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e126d73b-57c2-4db6-a74e-056fc325fcaf None admin] PATH_INFO: `/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.444983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e126d73b-57c2-4db6-a74e-056fc325fcaf None admin] RBAC: Authorizing `identity:get_project(project_id=3660441d6e5e46a6a5a5a35e570c4c6b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:51.447191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e126d73b-57c2-4db6-a74e-056fc325fcaf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:51.448132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 90/184] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/projects/3660441d6e5e46a6a5a5a35e570c4c6b => generated 312 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:51.453511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-075b59ce-5fb5-431b-8519-ac8a98c18322 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:51.454129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.454423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.454701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.454805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-075b59ce-5fb5-431b-8519-ac8a98c18322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:51.471972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:51.473206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:51.525318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.530886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 95/185] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:51.532326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993d95b8-d562-481c-bf0e-8e3e04943894 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.532618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993d95b8-d562-481c-bf0e-8e3e04943894 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.532924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993d95b8-d562-481c-bf0e-8e3e04943894 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.534190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 96/186] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:51.534936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993d95b8-d562-481c-bf0e-8e3e04943894 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.549845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 91/187] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:51.554153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.646780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993d95b8-d562-481c-bf0e-8e3e04943894 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.653634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303ca32-0b49-4f8b-bf9e-958de1b5b5ce None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.669001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.681077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.681733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.682008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.682276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.683016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:51.683316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:51.684730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559366df-918e-4fa8-b11c-805ed9f6e039 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:51.690904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.691554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.691837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.692114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.692620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 97/188] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:51.692980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=3660441d6e5e46a6a5a5a35e570c4c6b, user.id=89aa4c239d794594b3270fddfe326280) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:51.693293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:51.693802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.694086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.694357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.694718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:51.701700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 92/189] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/role_assignments?user.id=89aa4c239d794594b3270fddfe326280&scope.project.id=3660441d6e5e46a6a5a5a35e570c4c6b&role.id=168377268ae14ad882f881d40dd39b5c => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:51.702397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.719735 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:51.721056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:51.786929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.809611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 98/190] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:51.809611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.817352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7823d6e-1174-48cf-b095-48f5db5fb8ba None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.859638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.860441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.860750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.861028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.862223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:51.863992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:51.865514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 93/191] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:51.866108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:51.923400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:51.945490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c7fd4a-1520-453d-ae84-59a9b363e2b6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.978905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:51.979699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:51.979979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:51.980250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:51.982242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b00b4b-5987-4469-83d8-f72be154c2e9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:51.990484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:51.998528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:51.999924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:52.000728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 99/192] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:52.001377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.018794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.019660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.019946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.020224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] PATH_INFO: `/v3/users/0da0947504584451abd14e3a888beb69` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.033426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] RBAC: Authorizing `identity:get_user(user_id=0da0947504584451abd14e3a888beb69)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.034901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.036555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 94/193] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:51 2022] GET /identity/v3/users/0da0947504584451abd14e3a888beb69 => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:52.037339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.055596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.104594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daebf6-a145-464f-883f-51a0f74f39ac None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.134546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01396fa2-0767-4122-aa76-87a9e513421f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.136035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.137970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.138747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.139024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.139290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.149234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:52.156940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:52.157851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 100/194] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:52.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.173221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.173936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.174167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.174436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.175141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.176560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.195062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 95/195] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:50:52 2022] POST /identity/v3/users => generated 297 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:52.195852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.285891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df318301-7cce-4e02-97be-c3d6df0714ae None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.301273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.306213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8e69e-0d69-42e0-a0f9-63824f23e33f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.331290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.332006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.332416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.332717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.333900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:52.334211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.335686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.345054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 101/196] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:52.345913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.350136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.350971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.351410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.351686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.382456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.434562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.436058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.471060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f238ff0-6748-4982-a531-12a7ca172d6d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.524433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.525397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.525689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.526822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.527610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:52.527983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.529443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.541365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 102/197] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:52.542506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.566413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 96/198] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:52.567680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.581109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.655278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.661815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0d563d-f9d6-4d04-b941-8abe6582a0c9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.699650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.700990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.701464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.701908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] PATH_INFO: `/v3/users/0da0947504584451abd14e3a888beb69` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.704078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] RBAC: Authorizing `identity:get_user(user_id=0da0947504584451abd14e3a888beb69)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.705728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.707847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 103/199] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/users/0da0947504584451abd14e3a888beb69 => generated 283 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:52.708678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.710345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0405d-26e3-431c-b826-5efe6acd3203 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.756447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.758386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.758849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.759297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] PATH_INFO: `/v3/users/0fcaa119b56d42a19ed02a44c602c5c1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.761251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] RBAC: Authorizing `identity:get_user(user_id=0fcaa119b56d42a19ed02a44c602c5c1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.762711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.764370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 97/200] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/users/0fcaa119b56d42a19ed02a44c602c5c1 => generated 289 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:52.767946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.827493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66ca0e-ea54-4aca-b31d-71fbeab226ac None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.829174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.877538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.878401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.878682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.878956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:52.880154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:52.881688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:52.883419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 104/201] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:52.884815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:52.922150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b5b6b8-9d07-4c2b-8b41-7295afe1b967 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:52.956265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:52.968822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:52.969703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:52.970365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:52.970644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.027818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f12e99-4083-414e-b80f-eff6f18dcda5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:53.051822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.053619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.066592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:53.067646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:53.068120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:53.068603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:53.162917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.165045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.168444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 98/202] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:53.169652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:53.198290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:53.273926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a457cebf-0fa9-43a0-a61a-3e1a295586ec None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:53.277661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 105/203] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:53.278621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:53.311369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:53.312425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:53.312931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:53.313398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.314419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=0da0947504584451abd14e3a888beb69) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:53.314935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.316572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.322570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 99/204] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:53 2022] GET /identity/v3/role_assignments?user.id=0da0947504584451abd14e3a888beb69&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:53.323603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:53.384220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:53.411971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5fa47c-b51d-4280-b2c1-e625c2a70e75 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:53.452959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:53.455230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:53.455774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:53.456294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.457849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.459550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.462597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 106/205] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:53 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:53.473846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:53.573030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:53.589882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00dafd-4322-4bfa-93c9-5e3362624bb8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:53.653827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:53.654687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:53.654965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:53.655274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.657079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.658517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.661883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 100/206] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:53 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:53.667543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:53.670263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b368b-eccb-4190-88d3-fabff7e5083a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:53.753323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:53.753993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:53.754280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:53.754558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.755402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] RBAC: Adding query filter params (role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=0da0947504584451abd14e3a888beb69) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:53.755715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.760725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb88660-c600-4252-91e3-cf3df198c8e9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.767936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 107/207] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:53 2022] GET /identity/v3/role_assignments?user.id=0da0947504584451abd14e3a888beb69&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 259 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:53.786520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9138180d-0f3b-4bce-bc3b-1ed03278477f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:53.836421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:53.845316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:53.845631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:53.845889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:53.846800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] RBAC: Adding query filter params (role.id=0dd145a4768f4049b471a6b9c1ce4562, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, user.id=0fcaa119b56d42a19ed02a44c602c5c1) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:53.847120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:53.848613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b2fcc2-a264-4da1-a746-32ff91c3808d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:53.855731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 101/208] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:53 2022] GET /identity/v3/role_assignments?user.id=0fcaa119b56d42a19ed02a44c602c5c1&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=0dd145a4768f4049b471a6b9c1ce4562 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:54.511066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675586e-6cfb-48d4-a5ca-e35e3042d669 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.511438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675586e-6cfb-48d4-a5ca-e35e3042d669 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.511631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675586e-6cfb-48d4-a5ca-e35e3042d669 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.513353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 108/209] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:54.518481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246de86f-fa9b-42a4-b791-d48154a3d34f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246de86f-fa9b-42a4-b791-d48154a3d34f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.519454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246de86f-fa9b-42a4-b791-d48154a3d34f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.537905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-246de86f-fa9b-42a4-b791-d48154a3d34f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:54.539426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-246de86f-fa9b-42a4-b791-d48154a3d34f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:54.558321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:54.610308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 102/210] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:54.622695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e6c30-e6d7-4a1b-a535-a370aa6c7c26 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.622978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e6c30-e6d7-4a1b-a535-a370aa6c7c26 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.623265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e6c30-e6d7-4a1b-a535-a370aa6c7c26 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.625327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 109/211] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:54.631361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.631777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.632164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.637282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e6c30-e6d7-4a1b-a535-a370aa6c7c26 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:54.640779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:54.650291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:54.651150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:54.744720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 103/212] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:54.756628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e6c30-e6d7-4a1b-a535-a370aa6c7c26 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:54.765089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:54.792723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:54.793822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.794304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.794936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.806310 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:54.814412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:54.815309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:54.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:54.818522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 110/213] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:54.824406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:54.881074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:54.890253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4170ec66-a0e1-4a81-8d83-f051bf536059 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:54.891103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:54.934160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f21e03-657a-4ecc-bdc4-8401b1daef5e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:54.944245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:54.945960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.946321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.946891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.948229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:54.950109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:54.957004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 104/214] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:54.958640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:54.978122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:54.978810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:54.979096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:54.979375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:54.980431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:54.980764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:54.982345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:54.998283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 111/215] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:55.000824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:55.091726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0af955-89b8-4bf4-ac88-a7a0fca4ccde None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:55.176647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ae564-25b1-45f4-9b5f-0c89c9772b43 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:55.184069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:55.185424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.185941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.186414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.187747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:55.190029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:55.204506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:55.217326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 105/216] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:50:54 2022] POST /identity/v3/users => generated 297 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:50:55.219726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:55.220606 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.220922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.221197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] PATH_INFO: `/v3/users/d66744d8a442463a874c553296c2235d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.238763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] RBAC: Authorizing `identity:get_user(user_id=d66744d8a442463a874c553296c2235d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:55.240242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:55.242330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 112/217] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:54 2022] GET /identity/v3/users/d66744d8a442463a874c553296c2235d => generated 297 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:55.247894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:55.258271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:55.363451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da0f627-49ac-45ff-b432-413b674ed8f3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:55.400685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d7bcda-fead-4107-a2f0-c957e6ec1792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:55.401369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7bcda-fead-4107-a2f0-c957e6ec1792 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.401658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7bcda-fead-4107-a2f0-c957e6ec1792 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.401938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7bcda-fead-4107-a2f0-c957e6ec1792 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.474232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d7bcda-fead-4107-a2f0-c957e6ec1792 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:55.475727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d7bcda-fead-4107-a2f0-c957e6ec1792 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:55.484874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:55.585464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 106/218] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:55.590889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:55.598162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eacf9e-493c-4252-908b-411fe6cbab4f None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.598492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eacf9e-493c-4252-908b-411fe6cbab4f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.598779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eacf9e-493c-4252-908b-411fe6cbab4f None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.600559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 107/219] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:55.606830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838df817-68b0-4575-bc04-15018152741d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.607119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838df817-68b0-4575-bc04-15018152741d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.607388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838df817-68b0-4575-bc04-15018152741d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.629520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-838df817-68b0-4575-bc04-15018152741d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:55.630090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-838df817-68b0-4575-bc04-15018152741d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:55.720847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 108/220] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:55.725716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aa7fec-45ac-49f3-84af-ae000987adba None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:55.743559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838df817-68b0-4575-bc04-15018152741d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:55.781653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:55.782555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.782840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.783166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:55.785870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:55.787590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 113/221] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:55 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:55.797618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:55.807109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:55.859725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:55.909085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838df817-68b0-4575-bc04-15018152741d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:55.972724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c31eb-c53b-45e8-8422-2ba4ed42fcc5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:55.979388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d51c7-cddd-4bc9-a08e-6db4b2c41466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:55.980201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d51c7-cddd-4bc9-a08e-6db4b2c41466 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:55.980490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d51c7-cddd-4bc9-a08e-6db4b2c41466 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:55.980799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d51c7-cddd-4bc9-a08e-6db4b2c41466 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:55.995111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d51c7-cddd-4bc9-a08e-6db4b2c41466 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-954d51c7-cddd-4bc9-a08e-6db4b2c41466 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:56.003085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:56.003987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:56.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 109/222] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:56.006029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa64b27-c0e0-419c-ad45-e78b1c625ade None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.006291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa64b27-c0e0-419c-ad45-e78b1c625ade None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.006576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa64b27-c0e0-419c-ad45-e78b1c625ade None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.008148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 110/223] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:56.012583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa64b27-c0e0-419c-ad45-e78b1c625ade None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.025135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.026428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.027146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.027594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.028914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=d66744d8a442463a874c553296c2235d) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:56.029923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:56.032657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd940138-3e91-473b-ac9a-13fc677acdfb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:56.041171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 114/224] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:55 2022] GET /identity/v3/role_assignments?user.id=d66744d8a442463a874c553296c2235d&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=168377268ae14ad882f881d40dd39b5c => generated 259 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:56.042662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.043098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.043545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.065616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:56.066884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:56.159265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:56.167640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa64b27-c0e0-419c-ad45-e78b1c625ade None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:56.194448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 115/225] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:56.207992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.208989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.209510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.209982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.211054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:56.211558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:56.213294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:56.217105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.224781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 111/226] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:56.229666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.314980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:56.363253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49e729-95e7-4021-af5c-8411d9eac6be None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:56.408241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.409142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.409478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.409696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] PATH_INFO: `/v3/users/0da0947504584451abd14e3a888beb69` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.412068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] RBAC: Authorizing `identity:get_user(user_id=0da0947504584451abd14e3a888beb69)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:56.413536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:56.415190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 112/227] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/users/0da0947504584451abd14e3a888beb69 => generated 283 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:56.424850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.453528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445d2254-ca7f-4fa3-8d3b-63c29da5da09 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:56.473269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:56.505975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.508189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.510211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.510725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.532423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:56.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:56.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:50:56.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:56.545918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 116/228] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:50:56.551956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.641333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741df15e-518a-444f-bd3f-2dbd963b5d03 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:56.693741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9f77ca-fe40-4720-a432-22e84cfa3028 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:56.709047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:56.730588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.731645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.732134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.732615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.734057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:56.734601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:56.736269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:56.746705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 117/229] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:56.751394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.752295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.752584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.752888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.758007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.758906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:56.763488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:56.766138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 113/230] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:56.773967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.880210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62104b8-0328-4900-8b73-fd35ff6a8971 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:56.946991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:56.947793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:56.948080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:56.948358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] PATH_INFO: `/v3/users/0da0947504584451abd14e3a888beb69` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:56.950337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] RBAC: Authorizing `identity:get_user(user_id=0da0947504584451abd14e3a888beb69)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:56.951797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:56.954226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 118/231] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/users/0da0947504584451abd14e3a888beb69 => generated 283 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:56.960658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:56.964455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:56.970462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a9ac3-45f4-4afa-8a0e-75c3b5aef248 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:57.022304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:57.042504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:57.043399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:57.043686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:57.043963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964/users/0da0947504584451abd14e3a888beb69/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:57.047014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] RBAC: Authorizing `identity:create_grant(project_id=9320bdfceba6447f80351a3d3e908964, user_id=0da0947504584451abd14e3a888beb69, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:57.048463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:57.085218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 114/232] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:56 2022] PUT /identity/v3/projects/9320bdfceba6447f80351a3d3e908964/users/0da0947504584451abd14e3a888beb69/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:57.116640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aeef36-b096-46eb-95de-a52e1eb40ac5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:57.153787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25b44d8-f41a-48c0-9fc1-a46d298fee3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:57.154654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25b44d8-f41a-48c0-9fc1-a46d298fee3d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:57.154941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25b44d8-f41a-48c0-9fc1-a46d298fee3d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:57.155218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25b44d8-f41a-48c0-9fc1-a46d298fee3d None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:57.156542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25b44d8-f41a-48c0-9fc1-a46d298fee3d None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:57.158115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25b44d8-f41a-48c0-9fc1-a46d298fee3d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:57.160130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 119/233] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:56 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:57.165842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:57.279240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53eee3-cdc4-4798-88ef-43d47b0320ee None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:57.322292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8409948a-9738-4eec-82e8-ca1ae163c676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:57.323202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8409948a-9738-4eec-82e8-ca1ae163c676 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:57.323488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8409948a-9738-4eec-82e8-ca1ae163c676 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:57.323763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8409948a-9738-4eec-82e8-ca1ae163c676 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964/users/0da0947504584451abd14e3a888beb69/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:57.326057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8409948a-9738-4eec-82e8-ca1ae163c676 None admin] RBAC: Authorizing `identity:create_grant(project_id=9320bdfceba6447f80351a3d3e908964, user_id=0da0947504584451abd14e3a888beb69, role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:57.327685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8409948a-9738-4eec-82e8-ca1ae163c676 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:57.343283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 115/234] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:57 2022] PUT /identity/v3/projects/9320bdfceba6447f80351a3d3e908964/users/0da0947504584451abd14e3a888beb69/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:50:57.819369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e71f638-0da1-4b09-b12c-5f4d267501ac None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:57.819717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e71f638-0da1-4b09-b12c-5f4d267501ac None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:57.819937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e71f638-0da1-4b09-b12c-5f4d267501ac None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:57.821363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 120/235] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:57.826655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:57.827101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:57.827314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:57.852843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:57.854053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:57.869139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:57.950991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 116/236] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:57.987975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e71f638-0da1-4b09-b12c-5f4d267501ac None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.110242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e71f638-0da1-4b09-b12c-5f4d267501ac None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:58.111723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:58.145882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1574b734-3f93-4a85-b17f-97b9e5994192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:58.146682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1574b734-3f93-4a85-b17f-97b9e5994192 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.146967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1574b734-3f93-4a85-b17f-97b9e5994192 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.147242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1574b734-3f93-4a85-b17f-97b9e5994192 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.158085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1574b734-3f93-4a85-b17f-97b9e5994192 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-1574b734-3f93-4a85-b17f-97b9e5994192 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:58.166001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:58.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:50:58.167673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 121/237] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:58.175220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.204909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5212a-6bde-4567-8d27-243e38ff0c00 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.205220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5212a-6bde-4567-8d27-243e38ff0c00 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.205467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5212a-6bde-4567-8d27-243e38ff0c00 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.206991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 122/238] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:50:58.211965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.212258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.212537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.236728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:58.237955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:58.321656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 123/239] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:50:58.352396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e5b2a7-08de-4062-9849-256bbf434e3e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:58.362253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.397686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447157d1-110b-4d54-93a1-328252896edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:58.398314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447157d1-110b-4d54-93a1-328252896edc None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.398603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447157d1-110b-4d54-93a1-328252896edc None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.398891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447157d1-110b-4d54-93a1-328252896edc None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.399632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447157d1-110b-4d54-93a1-328252896edc None admin] RBAC: Adding query filter params (name=reader) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:58.399939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447157d1-110b-4d54-93a1-328252896edc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:58.401371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447157d1-110b-4d54-93a1-328252896edc None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:58.416640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 117/240] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:58.423656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447157d1-110b-4d54-93a1-328252896edc None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.432603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:58.478360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:58.489534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92bcb42-11dc-417c-b40b-d07458742c4a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:58.527814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8e1008-5a72-4f47-b24a-7acd796e6ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:58.529125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8e1008-5a72-4f47-b24a-7acd796e6ad1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.529407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8e1008-5a72-4f47-b24a-7acd796e6ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.529681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8e1008-5a72-4f47-b24a-7acd796e6ad1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.530864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8e1008-5a72-4f47-b24a-7acd796e6ad1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:58.532298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8e1008-5a72-4f47-b24a-7acd796e6ad1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:58.534248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 124/241] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:50:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:58.535554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23fcdc-a388-4cb2-aac8-0089d3e4c5b6 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.535835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23fcdc-a388-4cb2-aac8-0089d3e4c5b6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.536107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23fcdc-a388-4cb2-aac8-0089d3e4c5b6 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.537605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 125/242] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:58.544815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.545359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.545829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.563532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:58.565166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:58.565472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447157d1-110b-4d54-93a1-328252896edc None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:58.605594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:58.606808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.607282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.607743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] PATH_INFO: `/v3/users/11736f6e36be40439c5c7f18b38e0dfc` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.623636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] RBAC: Authorizing `identity:get_user(user_id=11736f6e36be40439c5c7f18b38e0dfc)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:58.626106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:58.627865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 118/243] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:58 2022] GET /identity/v3/users/11736f6e36be40439c5c7f18b38e0dfc => generated 297 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:58.632795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.647546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 126/244] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:50:58.648752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.719882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:58.765361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:58.767504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa013b-d7ec-44aa-81f5-272d9653e827 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:58.777572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcdd88b-02a6-4e01-a671-b37f71cf5278 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:58.806678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:58.807360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.807645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.807922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.808641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:58.810303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:58.823221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:58.823796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:58.824130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:58.824354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:58.834433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 119/245] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:50:58 2022] POST /identity/v3/users => generated 293 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:50:58.841675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:58.927557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:58.929628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:59.044818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:59.057566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 127/246] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:50:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:50:59.065984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.103551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fef9f02-9723-48f3-81ec-8ab4d8482028 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.125503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:59.141365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.142173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.142454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.142734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.154445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:59.162804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:59.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:59.166864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 120/247] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:59.177778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.178302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dded8a7e-6305-4e84-bd00-80cae3a5be10 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.215867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.216833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.217170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.217443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.219202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:59.220751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:59.222899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 128/248] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:59.228583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.294622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99203ece-3ea5-45cb-898e-41a4ac95a9a0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.296636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:59.335960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.336636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.336946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.337221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.338091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] RBAC: Adding query filter params (name=member) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:59.338501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:59.339900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:59.345103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c190b26-a823-4959-955d-c7546a22450c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.354131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 121/249] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:59.355480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.382490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.383257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.383546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.383926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.384792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] RBAC: Adding query filter params (role.id=0dd145a4768f4049b471a6b9c1ce4562, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=11736f6e36be40439c5c7f18b38e0dfc) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:50:59.385108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:59.386818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:59.394128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 129/250] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/role_assignments?user.id=11736f6e36be40439c5c7f18b38e0dfc&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=0dd145a4768f4049b471a6b9c1ce4562 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:59.404729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:59.471193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464000e3-11ba-4be4-8e13-626d754b9d41 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.508974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616263f6-e707-4319-996a-6c5891e98ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.509859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616263f6-e707-4319-996a-6c5891e98ec8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.510144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616263f6-e707-4319-996a-6c5891e98ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.510419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616263f6-e707-4319-996a-6c5891e98ec8 None admin] PATH_INFO: `/v3/users/d66744d8a442463a874c553296c2235d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.511668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616263f6-e707-4319-996a-6c5891e98ec8 None admin] RBAC: Authorizing `identity:get_user(user_id=d66744d8a442463a874c553296c2235d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:59.513189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616263f6-e707-4319-996a-6c5891e98ec8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:59.515449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 122/251] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/users/d66744d8a442463a874c553296c2235d => generated 297 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:50:59.521324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.564481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc00ff61-6e1e-4d92-9d64-989f091d436e None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.564820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc00ff61-6e1e-4d92-9d64-989f091d436e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.565114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc00ff61-6e1e-4d92-9d64-989f091d436e None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.566815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 123/252] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:59.572847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.573135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.573402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.596102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:50:59.597335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:50:59.661918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c84f86-9ecb-489f-97ad-ef6288d51d31 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.681919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 124/253] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:50:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 19 10:50:59.690201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:59.703591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.704798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.705273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.705758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.706944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.707391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:50:59.709376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:50:59.712874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 130/254] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:50:59.719506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:50:59.761037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:50:59.845160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93351148-dcee-48b5-95a8-c027bb1174c2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.892097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f302ddd9-600e-494f-9ff2-c116b3a15f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:50:59.893495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302ddd9-600e-494f-9ff2-c116b3a15f0b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.895078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302ddd9-600e-494f-9ff2-c116b3a15f0b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.895643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302ddd9-600e-494f-9ff2-c116b3a15f0b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.916433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f302ddd9-600e-494f-9ff2-c116b3a15f0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-f302ddd9-600e-494f-9ff2-c116b3a15f0b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:50:59.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:50:59.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:50:59.928148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:50:59.930978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 125/255] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:50:59.932513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f95518b-e53c-4a60-8120-4b2cdfc6f9be None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:50:59.932751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4bf691-43b0-4945-8f8e-95d43688eb98 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:50:59.933223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4bf691-43b0-4945-8f8e-95d43688eb98 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:50:59.933690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4bf691-43b0-4945-8f8e-95d43688eb98 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:50:59.936304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 126/256] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:50:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:50:59.937789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4bf691-43b0-4945-8f8e-95d43688eb98 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:00.009634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bf9eb6-7c9a-44f2-9180-e2c8caf15af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:00.014035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bf9eb6-7c9a-44f2-9180-e2c8caf15af9 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.014273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bf9eb6-7c9a-44f2-9180-e2c8caf15af9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.014791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bf9eb6-7c9a-44f2-9180-e2c8caf15af9 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964/users/d66744d8a442463a874c553296c2235d/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.023745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bf9eb6-7c9a-44f2-9180-e2c8caf15af9 None admin] RBAC: Authorizing `identity:create_grant(project_id=9320bdfceba6447f80351a3d3e908964, user_id=d66744d8a442463a874c553296c2235d, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:00.025439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bf9eb6-7c9a-44f2-9180-e2c8caf15af9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:00.054896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 131/257] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:50:59 2022] PUT /identity/v3/projects/9320bdfceba6447f80351a3d3e908964/users/d66744d8a442463a874c553296c2235d/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:00.058118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.059885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.060350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.103202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:00.104626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:00.122720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:00.130669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4bf691-43b0-4945-8f8e-95d43688eb98 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:00.169079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:00.201233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 132/258] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:00.204219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:00.205028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.205312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.205774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.206691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:00.207140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:00.209192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:00.227211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:00.229056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 127/259] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:50:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:00.237332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:00.382510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9292f72-462d-4d18-83b9-cfae1df4d3a2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:00.421081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:00.422197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.422420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cb727f-b089-4a22-8be9-47027ed728bd None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:00.422796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.423206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] PATH_INFO: `/v3/users/0da0947504584451abd14e3a888beb69` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.424571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] RBAC: Authorizing `identity:get_user(user_id=0da0947504584451abd14e3a888beb69)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:00.426228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:00.428162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 128/260] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:00 2022] GET /identity/v3/users/0da0947504584451abd14e3a888beb69 => generated 283 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:00.433800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:00.466760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:00.513640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:00.514634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.515589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.515965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.544575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc58658-5592-4922-b37b-8ebb2d1b7ebf None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:00.546576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:00.556043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:00.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:00.564519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 133/261] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:00.586826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:00.604919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:00.605781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.606227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.606637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.659107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:00.721264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:00.722762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:00.741333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0425895e-a090-4e1e-91d8-072a290b1e11 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:00.785791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:00.795443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:00.797070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:00.797558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:00.798021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:00.798982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:00.799478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:00.801591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:00.811373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 134/262] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:00.817602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:00.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 129/263] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 19 10:51:00.949433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:01.079534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cd6b4b-b50d-4759-ab67-8872c5826116 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:01.079721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6dbed-32b7-4801-b274-7c998e3d005e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:01.081579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:01.143828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:01.150853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:01.151736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:01.152023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:01.152344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] PATH_INFO: `/v3/users/0da0947504584451abd14e3a888beb69` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:01.154038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:01.154902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:01.155184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:01.156006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:01.157259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] RBAC: Authorizing `identity:get_user(user_id=0da0947504584451abd14e3a888beb69)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:01.158711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:01.161166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:01.162626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:01.164367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 130/264] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:00 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:01.164639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 135/265] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:00 2022] GET /identity/v3/users/0da0947504584451abd14e3a888beb69 => generated 283 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:01.169920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:01.171182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:01.365931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca9733-869e-42bc-9bf0-24fc0711db81 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:01.414607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e75aa0e-0fcb-4566-bcbd-fcee91e7a12b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:01.431437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:01.432370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:01.432771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:01.433153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:01.434245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=0da0947504584451abd14e3a888beb69) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:01.434656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:01.437184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:01.445128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 131/266] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:01 2022] GET /identity/v3/role_assignments?user.id=0da0947504584451abd14e3a888beb69&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:01.468676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:01.469623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:01.470077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:01.470497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:01.517216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:01.547928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:01.549798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:01.665709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 136/267] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:01.672263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:01.753424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:01.793420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5aa2a-d058-403c-a47e-ccb3ae6bfb73 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:01.829953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190ae276-81dd-4109-99da-4e44f549c366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:01.830829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190ae276-81dd-4109-99da-4e44f549c366 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:01.831115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190ae276-81dd-4109-99da-4e44f549c366 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:01.831392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190ae276-81dd-4109-99da-4e44f549c366 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:01.832608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190ae276-81dd-4109-99da-4e44f549c366 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:01.834111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190ae276-81dd-4109-99da-4e44f549c366 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:01.836140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 132/268] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:01 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:01.842216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:01.936317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:01.971515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b20ea38-7e22-424b-adc2-5e0ecec9e34a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:02.018795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:02.019473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.019749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.020022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.020887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] RBAC: Adding query filter params (role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=0da0947504584451abd14e3a888beb69) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:02.021199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:02.022638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01cb74b-b445-4607-8a12-0793745aaf36 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:02.029775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 137/269] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:01 2022] GET /identity/v3/role_assignments?user.id=0da0947504584451abd14e3a888beb69&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:02.407187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3ae8b-2a44-4d4b-8cd2-46c497abf575 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.408181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3ae8b-2a44-4d4b-8cd2-46c497abf575 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.408474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3ae8b-2a44-4d4b-8cd2-46c497abf575 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.410043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 133/270] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:02.415414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cbf71c-2cf4-4043-81a5-500c159023f5 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.416212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cbf71c-2cf4-4043-81a5-500c159023f5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.416679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cbf71c-2cf4-4043-81a5-500c159023f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.434335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-01cbf71c-2cf4-4043-81a5-500c159023f5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:02.435881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-01cbf71c-2cf4-4043-81a5-500c159023f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:02.508298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 138/271] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:02.532046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3ae8b-2a44-4d4b-8cd2-46c497abf575 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:02.543241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ec49c-6b83-443c-bcec-762cf56cbe0e None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.543517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ec49c-6b83-443c-bcec-762cf56cbe0e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.543788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ec49c-6b83-443c-bcec-762cf56cbe0e None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.545517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 139/272] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:02.550530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09561b0a-b483-4ea2-aa29-d992952e6350 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.550812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09561b0a-b483-4ea2-aa29-d992952e6350 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.551085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09561b0a-b483-4ea2-aa29-d992952e6350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.568329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-09561b0a-b483-4ea2-aa29-d992952e6350 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:02.569593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-09561b0a-b483-4ea2-aa29-d992952e6350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:02.640993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:02.651335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 140/273] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:02.652846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90947719-567a-4029-98d0-76aae3a28a18 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.653132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90947719-567a-4029-98d0-76aae3a28a18 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.653403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90947719-567a-4029-98d0-76aae3a28a18 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.654798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 141/274] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:02.660667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.661151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.661562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.665650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3ae8b-2a44-4d4b-8cd2-46c497abf575 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:02.678346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:02.679715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:02.700485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:02.707554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:02.708349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.708632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.708935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.720624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:02.729483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:02.730520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:02.731355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 134/275] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:02.732038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:02.764718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 142/276] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:02.764718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:02.852481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e1d1b5-3c3d-4d96-ad0b-f4d7eb02fb05 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:02.862875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504b4d12-30a2-45e4-9111-4cd2bec94c3f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:02.885764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:02.886372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:02.886540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.886820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.887097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.888587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:02.890872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:02.892508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 135/277] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:02.893300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:02.901582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:02.902254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.902564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.902840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.903587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:02.903895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:02.905329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5064f15-3c7c-4d24-aabc-9b7f06461f33 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:02.913769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 143/278] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:02.915444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc70a67-fe03-407e-b3a4-d1a8fdb021be None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:02.916048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc70a67-fe03-407e-b3a4-d1a8fdb021be None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:02.916330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc70a67-fe03-407e-b3a4-d1a8fdb021be None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:02.919111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 144/279] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:02.919719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc70a67-fe03-407e-b3a4-d1a8fdb021be None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:02.953184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:02.993737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e40c2-149d-43ba-a9c3-36fa06a093e2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.021320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc70a67-fe03-407e-b3a4-d1a8fdb021be None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.026753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.027577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.027871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.028146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.038268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:03.045867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:03.046849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:03.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 136/280] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:03.048311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.054810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e15cef-8797-4ad7-8177-9a42e9a92b71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.055474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e15cef-8797-4ad7-8177-9a42e9a92b71 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.055764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e15cef-8797-4ad7-8177-9a42e9a92b71 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.056036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e15cef-8797-4ad7-8177-9a42e9a92b71 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.056764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e15cef-8797-4ad7-8177-9a42e9a92b71 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.058207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e15cef-8797-4ad7-8177-9a42e9a92b71 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.079449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 145/281] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:02 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:03.080719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.081005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.081273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.104594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:03.105861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:03.145647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:03.164415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635e06a5-cf25-49a7-b527-53e37be23723 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.184747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 146/282] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:03.184747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.200308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:03.225986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.227434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.227720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.228001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] PATH_INFO: `/v3/users/11736f6e36be40439c5c7f18b38e0dfc` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.229673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] RBAC: Authorizing `identity:get_user(user_id=11736f6e36be40439c5c7f18b38e0dfc)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.231555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.233481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 137/283] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/users/11736f6e36be40439c5c7f18b38e0dfc => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:03.235200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.295607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e9ffd4-3d98-44e3-afd7-d40e2a207b1b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.340744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.341518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.341863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.342206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.343080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:03.343446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.346038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.357036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 147/284] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:03.359636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.375316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b72857-2b98-4e53-a4c6-fa27a9c1d395 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.402338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:03.418328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.419045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.419346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.419606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.430769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:03.447397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:03.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:03.449385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 138/285] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:03.450039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.495566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd95945-b2d4-42eb-bd48-66a33a5ffbb3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.503540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:03.538688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.540122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.540822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.541362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.542878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.545172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.548113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 148/286] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:03.549050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.568545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00ccd8-b9f1-4e6b-acc7-dcdc07b6efc8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.606739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.607618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.607908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.608187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] PATH_INFO: `/v3/users/77a04f64cd7b464ab53130c7293b9f55` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.629257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] RBAC: Authorizing `identity:get_user(user_id=77a04f64cd7b464ab53130c7293b9f55)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.632052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.634488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 139/287] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/users/77a04f64cd7b464ab53130c7293b9f55 => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:03.636309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.669226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b263db8-609c-4120-9642-7b56e0ad86fb None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.670870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:03.713079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.713752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.714097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.714498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.715397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:03.715712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.717139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.726827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 149/288] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:03.732247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.764569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91d541-414e-422c-b19a-4469950483ea None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.767949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:03.812024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.813224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.813512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.813791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964/users/11736f6e36be40439c5c7f18b38e0dfc/roles/0dd145a4768f4049b471a6b9c1ce4562` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.816641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] RBAC: Authorizing `identity:create_grant(project_id=9320bdfceba6447f80351a3d3e908964, user_id=11736f6e36be40439c5c7f18b38e0dfc, role_id=0dd145a4768f4049b471a6b9c1ce4562)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.818425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.836574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 140/289] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:03 2022] PUT /identity/v3/projects/9320bdfceba6447f80351a3d3e908964/users/11736f6e36be40439c5c7f18b38e0dfc/roles/0dd145a4768f4049b471a6b9c1ce4562 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:03.841681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:03.880035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e6d908-7527-454d-862b-18d86d5d4ade None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:03.959359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:03.960032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:03.960315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:03.960588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:03.961479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, user.id=77a04f64cd7b464ab53130c7293b9f55, scope.system=all) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:03.961790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:03.963287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:03.979114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 150/290] 10.222.0.33 () {56 vars in 1308 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/role_assignments?user.id=77a04f64cd7b464ab53130c7293b9f55&scope.system=all&role.id=168377268ae14ad882f881d40dd39b5c => generated 226 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:04.035025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a5334-f1f5-417b-837a-d3a46fb31b16 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:04.074538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:04.075427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:04.075707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:04.076039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] PATH_INFO: `/v3/users/d66744d8a442463a874c553296c2235d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:04.077864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] RBAC: Authorizing `identity:get_user(user_id=d66744d8a442463a874c553296c2235d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:04.079331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:04.081374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 141/291] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:03 2022] GET /identity/v3/users/d66744d8a442463a874c553296c2235d => generated 297 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:04.086969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:04.118743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:04.200566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486e365-b026-445a-87af-b9071aef75c3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:04.237039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:04.237689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:04.238383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:04.238659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:04.308967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:04.310438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:04.346872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:04.448434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 151/292] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:04.454589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:04.509926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:04.598508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20622998-313f-45bc-942e-836063cb4d56 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:04.640805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07b47da-5cf3-4622-ba94-4dea7f6a1294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:04.642056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07b47da-5cf3-4622-ba94-4dea7f6a1294 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:04.642333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07b47da-5cf3-4622-ba94-4dea7f6a1294 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:04.642603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07b47da-5cf3-4622-ba94-4dea7f6a1294 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:04.649414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07b47da-5cf3-4622-ba94-4dea7f6a1294 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:04.650991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07b47da-5cf3-4622-ba94-4dea7f6a1294 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:04.652724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 142/293] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:04 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:04.658721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:04.785013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292e42b-ee84-4698-b7a2-46901c59137c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:04.819015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:04.832890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:04.833550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:04.833833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:04.834109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:04.834943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=d66744d8a442463a874c553296c2235d) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:04.835249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:04.836690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4196286d-a47d-4697-afdd-9314ab4339e0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:04.849603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 152/294] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:04 2022] GET /identity/v3/role_assignments?user.id=d66744d8a442463a874c553296c2235d&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=168377268ae14ad882f881d40dd39b5c => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:05.340907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ee669-ce0a-4f09-88c2-863eddf99425 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.340907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ee669-ce0a-4f09-88c2-863eddf99425 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.340907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ee669-ce0a-4f09-88c2-863eddf99425 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.341817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 143/295] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:05.348488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0dfe96-1d82-4009-ba53-e2a925a3db5b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.348804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0dfe96-1d82-4009-ba53-e2a925a3db5b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.349070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0dfe96-1d82-4009-ba53-e2a925a3db5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.371299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ce0dfe96-1d82-4009-ba53-e2a925a3db5b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:05.372499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ce0dfe96-1d82-4009-ba53-e2a925a3db5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:05.450315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 153/296] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:05.481592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434ee669-ce0a-4f09-88c2-863eddf99425 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:05.544586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24fee67-88bc-41c8-a094-43b00ab8a128 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.544954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24fee67-88bc-41c8-a094-43b00ab8a128 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.545160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24fee67-88bc-41c8-a094-43b00ab8a128 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.546728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 154/297] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:05.551663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.551950 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.552223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.574001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:05.575675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:05.600120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:05.604580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434ee669-ce0a-4f09-88c2-863eddf99425 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:05.642942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:05.644775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.645278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.645748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.661236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:05.664148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 155/298] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:05.670380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:05.671266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:05.672085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:05.674674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 144/299] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:05.679986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:05.687635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:05.737446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:05.786871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afcaed-341b-4ce4-a264-75028348f6e0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:05.792653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f267c45c-ca76-413a-9834-06a8179da981 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:05.821285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502f911d-d057-43c1-a043-893181bec658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:05.821946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502f911d-d057-43c1-a043-893181bec658 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.822581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502f911d-d057-43c1-a043-893181bec658 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.822856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502f911d-d057-43c1-a043-893181bec658 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.823709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502f911d-d057-43c1-a043-893181bec658 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:05.824015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502f911d-d057-43c1-a043-893181bec658 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:05.825579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502f911d-d057-43c1-a043-893181bec658 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:05.830281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:05.831437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.831923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:05.832394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:05.833802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:05.834926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 156/300] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:05.835979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:05.842284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502f911d-d057-43c1-a043-893181bec658 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:05.842624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 145/301] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:05.843771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:05.875087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:05.955380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502f911d-d057-43c1-a043-893181bec658 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:05.956236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8daa37-71ec-480a-924c-9ac72e36088f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:05.972324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:05.998047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059cc82f-97a8-47d3-8998-17f6bb419531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:05.998277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:05.998945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:05.999759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.000036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] PATH_INFO: `/v3/users/30d3570d560140eba4b3cfc9c86e5d6d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.000377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059cc82f-97a8-47d3-8998-17f6bb419531 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.000890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059cc82f-97a8-47d3-8998-17f6bb419531 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.001349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059cc82f-97a8-47d3-8998-17f6bb419531 None admin] PATH_INFO: `/v3/groups` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.002407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059cc82f-97a8-47d3-8998-17f6bb419531 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:06.004891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059cc82f-97a8-47d3-8998-17f6bb419531 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:06.017262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] RBAC: Authorizing `identity:get_user(user_id=30d3570d560140eba4b3cfc9c86e5d6d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:06.020218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:06.027458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 157/302] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:05 2022] GET /identity/v3/users/30d3570d560140eba4b3cfc9c86e5d6d => generated 293 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:06.028744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:06.029099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 146/303] 10.222.0.33 () {60 vars in 1134 bytes} [Tue Jul 19 10:51:05 2022] POST /identity/v3/groups => generated 229 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:06.146246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ecb913-d84a-4344-8ca7-1a6ffad83cb7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:06.185335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:06.186407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.187434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.187896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.188937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] RBAC: Adding query filter params (role.id=0dd145a4768f4049b471a6b9c1ce4562, user.id=30d3570d560140eba4b3cfc9c86e5d6d, scope.system=all) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:06.189476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:06.191115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f393043f-f346-4cec-b34f-f41d06d5a627 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:06.197184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 158/304] 10.222.0.33 () {56 vars in 1308 bytes} [Tue Jul 19 10:51:06 2022] GET /identity/v3/role_assignments?user.id=30d3570d560140eba4b3cfc9c86e5d6d&scope.system=all&role.id=0dd145a4768f4049b471a6b9c1ce4562 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:06.380569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d13822d-faa0-4390-8819-cf1fc34fe321 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.380941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d13822d-faa0-4390-8819-cf1fc34fe321 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.381171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d13822d-faa0-4390-8819-cf1fc34fe321 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.382548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 147/305] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:06.387669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29be2ed9-c2bd-43de-abb2-513b8fa7961d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.387960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29be2ed9-c2bd-43de-abb2-513b8fa7961d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.388239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29be2ed9-c2bd-43de-abb2-513b8fa7961d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.406161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-29be2ed9-c2bd-43de-abb2-513b8fa7961d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:06.407265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-29be2ed9-c2bd-43de-abb2-513b8fa7961d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:06.438384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:06.484657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 159/306] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:06.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67424bc4-98c5-4d5c-aa0c-c3759d14df75 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.493638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67424bc4-98c5-4d5c-aa0c-c3759d14df75 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.493821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67424bc4-98c5-4d5c-aa0c-c3759d14df75 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.495181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 148/307] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:06.500339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.500632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.500959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.506340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67424bc4-98c5-4d5c-aa0c-c3759d14df75 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:06.518085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:06.519272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:06.607825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 160/308] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:06.614199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:06.628330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67424bc4-98c5-4d5c-aa0c-c3759d14df75 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:06.633855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:06.672479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:06.673752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.674353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.674843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.687703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:06.696779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:06.697671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:06.698442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:06.701396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 149/309] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:06.706666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:06.746877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:06.755986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc3dcb6-7a64-4f1a-8d6e-1f6eb570b604 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:06.794570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:06.795718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.796846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.797352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.809353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:06.810529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d749b-8876-4935-9b20-ef09688b8a10 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:06.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:06.818941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:06.819757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:06.827040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 161/310] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:06.828647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:06.844479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:06.845174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.845455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.845728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.846478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:06.846781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:06.848156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:06.861990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 150/311] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:06.862970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:06.875596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:06.956776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9153e04-a61d-4853-b59b-9534002582ee None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:06.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57d9c1a-4637-4069-bec8-89de2de3ccc5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:06.994989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:06.996091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:06.996573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:06.997180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:06.998137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:06.998651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.000279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.006091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:07.006919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:07.007203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:07.007477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] PATH_INFO: `/v3/users/77a04f64cd7b464ab53130c7293b9f55` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:07.008732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] RBAC: Authorizing `identity:get_user(user_id=77a04f64cd7b464ab53130c7293b9f55)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.010284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.015627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 151/313] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:06 2022] GET /identity/v3/users/77a04f64cd7b464ab53130c7293b9f55 => generated 293 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:07.015921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 162/313] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:07.017669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:07.017791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:07.041308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:07.106830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:07.130020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884066b2-a75e-44b4-8be2-edd7ec932f1d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:07.157357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221ebea1-cb97-4bec-9f5c-2d0322755c87 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:07.166991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:07.167866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:07.168145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:07.168417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] PATH_INFO: `/v3/users/11736f6e36be40439c5c7f18b38e0dfc` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:07.169782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] RBAC: Authorizing `identity:get_user(user_id=11736f6e36be40439c5c7f18b38e0dfc)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.171292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.177696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 152/314] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:07 2022] GET /identity/v3/users/11736f6e36be40439c5c7f18b38e0dfc => generated 297 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:07.180022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:07.203604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:07.204394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:07.204674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:07.204972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] PATH_INFO: `/v3/system/users/77a04f64cd7b464ab53130c7293b9f55/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:07.206454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=77a04f64cd7b464ab53130c7293b9f55, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.208087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.220560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 163/315] 10.222.0.33 () {58 vars in 1334 bytes} [Tue Jul 19 10:51:07 2022] PUT /identity/v3/system/users/77a04f64cd7b464ab53130c7293b9f55/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 10:51:07.301970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97237c2d-f2a7-4137-a0ec-9ad44bd34a0b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:07.345225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:07.346277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:07.346764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:07.347240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:07.367024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:07.428178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.429670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.533627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 153/316] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:07.539902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:07.625368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:07.644747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c153af8-4416-4c21-8450-65368adb87cb None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:07.685755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8805f65-07f4-4cdc-a9a6-5fca0ae4e86f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:07.686613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8805f65-07f4-4cdc-a9a6-5fca0ae4e86f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:07.686930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8805f65-07f4-4cdc-a9a6-5fca0ae4e86f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:07.687209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8805f65-07f4-4cdc-a9a6-5fca0ae4e86f None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:07.688409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8805f65-07f4-4cdc-a9a6-5fca0ae4e86f None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.689916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8805f65-07f4-4cdc-a9a6-5fca0ae4e86f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.691926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 164/317] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:07 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:07.697874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:07.748683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:07.805045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f68b5f-1fbf-4ef6-9d7e-357a5bb1ff3f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:07.845116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:07.845768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:07.846048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:07.846319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:07.847149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] RBAC: Adding query filter params (role.id=0dd145a4768f4049b471a6b9c1ce4562, scope.project.id=9320bdfceba6447f80351a3d3e908964, user.id=11736f6e36be40439c5c7f18b38e0dfc) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:07.847457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:07.848905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40927940-7b58-4cb0-b244-b39ac91cb6f0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:07.855090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 154/318] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:07 2022] GET /identity/v3/role_assignments?user.id=11736f6e36be40439c5c7f18b38e0dfc&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=0dd145a4768f4049b471a6b9c1ce4562 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:08.133361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73bafd1-ccf3-41c6-a896-e70e9b5b1aaa None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.133838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73bafd1-ccf3-41c6-a896-e70e9b5b1aaa None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.133838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73bafd1-ccf3-41c6-a896-e70e9b5b1aaa None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.135351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 165/319] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:08.140900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87779277-2d13-4766-89e6-46df42baca0d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.141191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87779277-2d13-4766-89e6-46df42baca0d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.141518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87779277-2d13-4766-89e6-46df42baca0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-87779277-2d13-4766-89e6-46df42baca0d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:08.158929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-87779277-2d13-4766-89e6-46df42baca0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:08.226805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 155/320] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:08.248668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73bafd1-ccf3-41c6-a896-e70e9b5b1aaa None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:08.357296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eb84a0-d1e4-4ea3-be7f-2b8361c3bada None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.357584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eb84a0-d1e4-4ea3-be7f-2b8361c3bada None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.357854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eb84a0-d1e4-4ea3-be7f-2b8361c3bada None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.360137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 156/321] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:08.364374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73bafd1-ccf3-41c6-a896-e70e9b5b1aaa None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:08.366828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.367193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.367484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.394704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:08.395864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:08.415666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:08.416446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.416748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.417017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.418160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:08.419584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:08.421910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 166/322] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:08.427037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:08.434015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:08.447795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:08.469660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 157/323] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:08.491746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:08.537985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831dc8fc-5da8-4e1b-9980-3dd9ee2e2b43 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:08.577006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:08.577689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.577969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.578238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] PATH_INFO: `/v3/groups` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.578911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:08.580331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:08.595991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4487f5-2145-4d16-b4ed-1713faab0e57 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:08.596665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 167/324] 10.222.0.33 () {60 vars in 1134 bytes} [Tue Jul 19 10:51:08 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:51:08.645955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:08.646699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.646987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.647265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.660327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:08.668479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:08.669443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:08.670251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 158/325] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:08.678308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:08.769972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:08.794433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576eefe5-e14a-48cd-b18b-eddd1b5632da None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:08.831414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:08.832079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:08.832360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:08.832632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:08.833429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:08.833727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:08.835109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:08.844000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 168/326] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:08 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:08.849578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:08.886529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:08.970491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a34deb-5e28-4fd7-9db3-7f9d0e8ac766 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:09.007528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba8b673-8d31-4646-a246-5239984070e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:09.008409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba8b673-8d31-4646-a246-5239984070e2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.008688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba8b673-8d31-4646-a246-5239984070e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.009002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba8b673-8d31-4646-a246-5239984070e2 None admin] PATH_INFO: `/v3/users/30d3570d560140eba4b3cfc9c86e5d6d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.010239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba8b673-8d31-4646-a246-5239984070e2 None admin] RBAC: Authorizing `identity:get_user(user_id=30d3570d560140eba4b3cfc9c86e5d6d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:09.011685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba8b673-8d31-4646-a246-5239984070e2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:09.013741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 159/327] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:08 2022] GET /identity/v3/users/30d3570d560140eba4b3cfc9c86e5d6d => generated 293 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:09.019619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:09.143028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8068c2ec-78be-4be8-9548-e532da899a08 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:09.183244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c82839-627c-4547-a7e9-56de39b682a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:09.184174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c82839-627c-4547-a7e9-56de39b682a3 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.184400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c82839-627c-4547-a7e9-56de39b682a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.184675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c82839-627c-4547-a7e9-56de39b682a3 None admin] PATH_INFO: `/v3/system/users/30d3570d560140eba4b3cfc9c86e5d6d/roles/0dd145a4768f4049b471a6b9c1ce4562` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.188178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c82839-627c-4547-a7e9-56de39b682a3 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=30d3570d560140eba4b3cfc9c86e5d6d, role_id=0dd145a4768f4049b471a6b9c1ce4562)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:09.189685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c82839-627c-4547-a7e9-56de39b682a3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:09.201505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 169/328] 10.222.0.33 () {58 vars in 1334 bytes} [Tue Jul 19 10:51:09 2022] PUT /identity/v3/system/users/30d3570d560140eba4b3cfc9c86e5d6d/roles/0dd145a4768f4049b471a6b9c1ce4562 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:09.348376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eacb90-78ea-4a6e-9fba-eb18d8de8bf1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.348798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eacb90-78ea-4a6e-9fba-eb18d8de8bf1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.348977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eacb90-78ea-4a6e-9fba-eb18d8de8bf1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.350671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 160/329] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:09.356378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.357007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.365975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:09.379525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:09.380673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:09.452827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 170/330] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:09.476059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eacb90-78ea-4a6e-9fba-eb18d8de8bf1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:09.591262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:09.596445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eacb90-78ea-4a6e-9fba-eb18d8de8bf1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:09.643416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:09.644223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.644506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.644813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.656450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:09.664433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:09.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:09.666141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 161/331] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:09.672474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:09.778612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21e1125-fa45-460c-a0b3-d2d5c8cc8a4e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:09.816146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:09.816889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.817138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.817414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.818163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:09.818470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:09.819848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:09.821228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:09.828577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 171/332] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:09.834194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:09.944723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da4370a-3a1f-4a5a-bfb9-05f449ee8a64 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:09.975195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7230662-1629-4d67-bb38-66b131e9c6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:09.976075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7230662-1629-4d67-bb38-66b131e9c6b8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:09.976356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7230662-1629-4d67-bb38-66b131e9c6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:09.976630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7230662-1629-4d67-bb38-66b131e9c6b8 None admin] PATH_INFO: `/v3/users/77a04f64cd7b464ab53130c7293b9f55` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:09.977865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7230662-1629-4d67-bb38-66b131e9c6b8 None admin] RBAC: Authorizing `identity:get_user(user_id=77a04f64cd7b464ab53130c7293b9f55)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:09.979324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7230662-1629-4d67-bb38-66b131e9c6b8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:09.981785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 162/333] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:09 2022] GET /identity/v3/users/77a04f64cd7b464ab53130c7293b9f55 => generated 293 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:09.987929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:10.102390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d591c06b-c9a4-4397-9919-dc055f34799b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:10.139822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:10.140525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:10.140839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:10.141118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:10.141961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] RBAC: Adding query filter params (role.id=168377268ae14ad882f881d40dd39b5c, user.id=77a04f64cd7b464ab53130c7293b9f55, scope.system=all) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:10.142274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:10.143706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacd7480-08c6-457e-9dca-5eb30507e5a8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:10.157437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 172/334] 10.222.0.33 () {56 vars in 1308 bytes} [Tue Jul 19 10:51:09 2022] GET /identity/v3/role_assignments?user.id=77a04f64cd7b464ab53130c7293b9f55&scope.system=all&role.id=168377268ae14ad882f881d40dd39b5c => generated 510 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:11.314154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf1eb4a-3d0a-48c8-bf97-c8ef8f6375c6 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:11.314629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf1eb4a-3d0a-48c8-bf97-c8ef8f6375c6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:11.314629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf1eb4a-3d0a-48c8-bf97-c8ef8f6375c6 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:11.316269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 163/335] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:11.321254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:11.322043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:11.322519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:11.345668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:11.347194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:11.408997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:11.428262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 173/336] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:11.467152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf1eb4a-3d0a-48c8-bf97-c8ef8f6375c6 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:11.500864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:11.591661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf1eb4a-3d0a-48c8-bf97-c8ef8f6375c6 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:11.637587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f5776d-0a70-445c-b765-7f3cef0e7ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:11.638766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f5776d-0a70-445c-b765-7f3cef0e7ca5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:11.639251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f5776d-0a70-445c-b765-7f3cef0e7ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:11.639722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f5776d-0a70-445c-b765-7f3cef0e7ca5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:11.655010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f5776d-0a70-445c-b765-7f3cef0e7ca5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-20f5776d-0a70-445c-b765-7f3cef0e7ca5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:11.663922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:11.664827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:11.665628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:11.672511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 164/337] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:11.673874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:11.797314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d906a-a31c-4611-9018-f94871050dc4 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:11.797947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d906a-a31c-4611-9018-f94871050dc4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:11.798445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d906a-a31c-4611-9018-f94871050dc4 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:11.799281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5af375f-fcbd-4d41-8202-f19987256acb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:11.800244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 165/338] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:11.806865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:11.807533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:11.807920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:11.842523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:11.843695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:11.870768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:11.871439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:11.871725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:11.871997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:11.872777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:11.873087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:11.875001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:11.888201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 174/339] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:11.895355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:11.990821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.010375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 166/340] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:12.065060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a527bf7b-b223-406a-9ffb-07b8cc6928bd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.103328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:12.105791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:12.106457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.106741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.107016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.139857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.181597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:12.183186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:12.249198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.293570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 175/341] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:12.301314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:12.406008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f109321-b6c9-42d8-8ee1-bd02f8173d73 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.416154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34baf30d-7d34-4829-a5d2-872ea2aebdea None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.465555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56411be6-be1d-4da7-9a31-e695132ba406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:12.466408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56411be6-be1d-4da7-9a31-e695132ba406 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.466692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56411be6-be1d-4da7-9a31-e695132ba406 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.466963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56411be6-be1d-4da7-9a31-e695132ba406 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.475642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aab68e1-5cbe-4f2a-8109-9133b1d8ce08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:12.476506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aab68e1-5cbe-4f2a-8109-9133b1d8ce08 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.476813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aab68e1-5cbe-4f2a-8109-9133b1d8ce08 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.477079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aab68e1-5cbe-4f2a-8109-9133b1d8ce08 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.478618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aab68e1-5cbe-4f2a-8109-9133b1d8ce08 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:12.480073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aab68e1-5cbe-4f2a-8109-9133b1d8ce08 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:12.481980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 176/342] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:12 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:12.482954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdafebf-8d94-42f1-b1ed-06a5ae35ce48 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.483234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdafebf-8d94-42f1-b1ed-06a5ae35ce48 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.483500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdafebf-8d94-42f1-b1ed-06a5ae35ce48 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.491980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56411be6-be1d-4da7-9a31-e695132ba406 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:12.497910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 177/343] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:12.498823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdafebf-8d94-42f1-b1ed-06a5ae35ce48 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-56411be6-be1d-4da7-9a31-e695132ba406 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:12.504520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:12.505500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:12.506280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 167/344] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 403 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:12.506932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.507501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.507501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.531748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:12.532989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:12.545043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.659944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 168/345] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:12.664732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:12.669261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.713774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdafebf-8d94-42f1-b1ed-06a5ae35ce48 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.755508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a89320c-6a33-4d49-9cc3-c60a15ba64d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:12.756808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a89320c-6a33-4d49-9cc3-c60a15ba64d1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.757432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a89320c-6a33-4d49-9cc3-c60a15ba64d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.757924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a89320c-6a33-4d49-9cc3-c60a15ba64d1 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.764681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a89320c-6a33-4d49-9cc3-c60a15ba64d1 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:12.766515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a89320c-6a33-4d49-9cc3-c60a15ba64d1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:12.768956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 178/346] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:12 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:12.770609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65cffb7-6b6f-4f77-ac9c-04ec7cec1525 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.771235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65cffb7-6b6f-4f77-ac9c-04ec7cec1525 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.771728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65cffb7-6b6f-4f77-ac9c-04ec7cec1525 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.773736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 179/347] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:12.774726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65cffb7-6b6f-4f77-ac9c-04ec7cec1525 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:12.780446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab62b84-d92e-41cf-80fb-6164f9a0d177 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.824786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:12.825505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.825785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.826056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.826960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:12.827271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:12.829486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:12.841651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 169/348] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:12.843808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:12.881197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.908914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65cffb7-6b6f-4f77-ac9c-04ec7cec1525 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.939760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:12.955962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d31907a-6afb-49e2-a749-4eb045205488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:12.958137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d31907a-6afb-49e2-a749-4eb045205488 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.958814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d31907a-6afb-49e2-a749-4eb045205488 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.959286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d31907a-6afb-49e2-a749-4eb045205488 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:12.971723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d31907a-6afb-49e2-a749-4eb045205488 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:12.979935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2a021-aeed-4e71-9720-21d67c01f6e5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6d31907a-6afb-49e2-a749-4eb045205488 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:12.981242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:12.982192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 19 10:51:12.983007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:12.985824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 180/349] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:12.987254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:12.992524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:12.993090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.013892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:13.015968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:13.026742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.027690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.028108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.028623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.029614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:13.030068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.032169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.041208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 170/350] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:12 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 260 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:13.042051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.139121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 181/351] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:13.141955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.188395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69842aa4-eef9-4d45-9d4d-bffae5ba8754 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.225326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:13.230457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:13.237269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.238239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.238717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.239199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.276614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc9020-cfe3-4ad3-830f-69ba6041c38c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.321633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.322317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.322602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.322879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.323629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:13.323935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.325423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1c1270-8bcb-416f-bdd4-624fb404b653 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.328155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.329939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.334696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 182/352] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:13.336070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d26bec-2fe6-4d36-a94c-818ed15c49af None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.336362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d26bec-2fe6-4d36-a94c-818ed15c49af None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.336641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d26bec-2fe6-4d36-a94c-818ed15c49af None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.337895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 183/353] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:13.338580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d26bec-2fe6-4d36-a94c-818ed15c49af None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.444324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 171/354] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:13.445821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.446750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d26bec-2fe6-4d36-a94c-818ed15c49af None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.476630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:13.481423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:13.483156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf6ef31-49d2-4a83-bcb8-94c6da55ee67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.483945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf6ef31-49d2-4a83-bcb8-94c6da55ee67 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.484230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf6ef31-49d2-4a83-bcb8-94c6da55ee67 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.484819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf6ef31-49d2-4a83-bcb8-94c6da55ee67 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.496809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf6ef31-49d2-4a83-bcb8-94c6da55ee67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9cf6ef31-49d2-4a83-bcb8-94c6da55ee67 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:13.506113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:13.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:13.507785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 184/355] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:13.508914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.512882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.513193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.531476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:13.532672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:13.561512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce28ac1c-84b1-4a8c-a9ba-a33fd223d08a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.602098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.602712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 185/356] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:13.603387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.604098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.604569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.605054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] PATH_INFO: `/v3/users/30d3570d560140eba4b3cfc9c86e5d6d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.606504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] RBAC: Authorizing `identity:get_user(user_id=30d3570d560140eba4b3cfc9c86e5d6d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.608248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.610886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 172/357] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/users/30d3570d560140eba4b3cfc9c86e5d6d => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:13.611844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.722754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359885f-2cfb-4f74-ab37-e206562525af None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.748894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:13.752668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:13.753444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f20b30-f5bb-4401-9e13-eb9a03e37743 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.772845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.773801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.774153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.774493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.775742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.777342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.780872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 186/358] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:13.780872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.792987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.793658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.793941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.794209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.794965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] RBAC: Adding query filter params (name=member) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:13.795266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.797453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7bafc9-4220-4b86-9dd7-51333c63bc69 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.811242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 173/359] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:13.811747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48e0010-d821-4c26-8d64-841af0f2e447 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.812661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48e0010-d821-4c26-8d64-841af0f2e447 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.812975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48e0010-d821-4c26-8d64-841af0f2e447 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.816732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 174/360] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:13.816732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48e0010-d821-4c26-8d64-841af0f2e447 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.904729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f7e21e-7759-474b-83b8-5626d2e7bf52 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.949267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48e0010-d821-4c26-8d64-841af0f2e447 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:13.954533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:13.955190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:13.955471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:13.955744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:13.956580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] RBAC: Adding query filter params (role.id=0dd145a4768f4049b471a6b9c1ce4562, user.id=30d3570d560140eba4b3cfc9c86e5d6d, scope.system=all) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:13.957633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:13.958357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:13.973034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 187/361] 10.222.0.33 () {56 vars in 1308 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/role_assignments?user.id=30d3570d560140eba4b3cfc9c86e5d6d&scope.system=all&role.id=0dd145a4768f4049b471a6b9c1ce4562 => generated 510 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:13.974117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:13.999856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.001041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.001467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.001758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.017868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:14.029056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:14.030047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:14.031361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:14.035641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 175/362] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:14.036936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.048463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.070934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.114713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeeb8b9-7af3-4d7c-9216-42916f1e4ab6 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.154321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae743fa6-96be-40e4-ac43-2529373ddd52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.155267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae743fa6-96be-40e4-ac43-2529373ddd52 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.155477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae743fa6-96be-40e4-ac43-2529373ddd52 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.156287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae743fa6-96be-40e4-ac43-2529373ddd52 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.158519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e0e494-7a92-4810-8ff6-d0b05d1e8782 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.158630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae743fa6-96be-40e4-ac43-2529373ddd52 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.160064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae743fa6-96be-40e4-ac43-2529373ddd52 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.166132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 188/363] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:13 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:14.167489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.167776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.168048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.189255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:14.190645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:14.197496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.198404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.198683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.198952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.270092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 189/364] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:14.270637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.289758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.291222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.320919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.330670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.406176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586e05f9-271b-4b39-a085-aa5aa79a251c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.422769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 176/365] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:14.423634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.450426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.451495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.451771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.452201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.453041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:14.453342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.455328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.469033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 190/366] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:14.469881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.533858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8565a1ce-43cb-4b13-beeb-c54deded7405 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.561776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.576855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.577686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.578034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.578377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.579286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:14.579665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.581198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.586958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 177/367] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=168377268ae14ad882f881d40dd39b5c => generated 260 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:14.587824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.590760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b1c14-502f-49d2-91bf-0608bf175de1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.599722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.638543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.639651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.639935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.640396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.655308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:14.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:14.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:14.666371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:14.666976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 191/368] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:14.668274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.730337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f83e689-d607-4718-8b89-e3349fa101d5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.773371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.774382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.774668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.774940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.778041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.779545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.783361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 178/369] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:14.784217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:14.799959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a70f1b1-7865-4b93-ab2f-b9dd841c1117 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.838074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.838780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.839068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.839346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.855922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.879658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:14.901810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d5483c-2bde-48d7-8b12-a97547071253 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:14.918777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.920249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.944020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:14.944739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:14.945039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:14.945322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:14.946086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:14.946582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:14.948061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:14.958855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 179/370] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:14.959816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.058199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 192/371] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:15.059823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.088887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316d8a17-9507-46e5-ac12-f31ccfcfc2a4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.111615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:15.138778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.140125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.140525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.141181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.143157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.144712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.146010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:15.148247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 180/372] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:14 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:15.149503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.189226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd060ff-a6d5-428f-b084-c879731032da None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.242421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.243853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.244143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.244418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.278840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52cfd00-9227-491c-9243-1b625f7f247f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.320505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.321628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.322119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.322583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.324120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.326091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.326721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.328321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.328767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 181/373] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:15.329965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.389393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:15.400951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:15.445256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 193/374] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:15.446429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.451577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495631ba-2cf1-43d1-8ea0-61dc243423d1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.502077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.502737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.503029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.503304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.504757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=9320bdfceba6447f80351a3d3e908964) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:15.505077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.506876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91980754-0c90-48c9-82a5-7bbd2bbe2376 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.514245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 182/375] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=168377268ae14ad882f881d40dd39b5c => generated 260 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:15.518359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11cc27e-13af-44ca-b076-77e77cece48b None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.518785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11cc27e-13af-44ca-b076-77e77cece48b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.519075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11cc27e-13af-44ca-b076-77e77cece48b None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.520896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 183/376] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:15.521692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11cc27e-13af-44ca-b076-77e77cece48b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.589541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba516d3-19da-468c-8c61-950617f92d8c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.649354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ab1ce-3ebf-49dc-97fc-c3c331b36716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.650231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ab1ce-3ebf-49dc-97fc-c3c331b36716 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.650512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ab1ce-3ebf-49dc-97fc-c3c331b36716 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.650782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ab1ce-3ebf-49dc-97fc-c3c331b36716 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.652964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ab1ce-3ebf-49dc-97fc-c3c331b36716 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.654446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ab1ce-3ebf-49dc-97fc-c3c331b36716 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.656849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 194/377] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:15.660166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.660166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.660166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.667770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11cc27e-13af-44ca-b076-77e77cece48b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.686175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:15.688039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:15.691431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:15.717927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.719379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.719664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.719940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] PATH_INFO: `/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.721286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] RBAC: Authorizing `identity:get_project(project_id=a1dc1eb7c79345caaeb191fa0f9c008f)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.722771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 184/378] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f => generated 344 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:15.725632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:15.725741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.785556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 195/379] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:15.785767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.852425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d17e22-896d-49f9-8325-cfa2439ecaf9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.898280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.899277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.899771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.900248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.901753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=9320bdfceba6447f80351a3d3e908964) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:15.902405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.904198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.917073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 185/380] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:15.918397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:15.925896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c65a909-b303-45d6-bc9e-8487924f8ea6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:15.978473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:15.980022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:15.980502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:15.981432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] PATH_INFO: `/v3/groups/dcca039955b044c4bbee4d33dc591efd` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:15.988602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] RBAC: Authorizing `identity:get_group(group_id=dcca039955b044c4bbee4d33dc591efd)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:15.991279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:15.993864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 196/381] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/groups/dcca039955b044c4bbee4d33dc591efd => generated 229 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:16.001117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:16.002388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:16.062474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c65602c-5c13-4629-a412-4ca69cf63883 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:16.080254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:16.105471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:16.106800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:16.107283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:16.107757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:16.124241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:16.135182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:16.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:16.136959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:16.139698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 186/382] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:16.146835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:16.258134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdc948e-d0ff-46ae-8477-9a9cc10c56b0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:16.269378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:16.299417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:16.300449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:16.300972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:16.301441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:16.302379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:16.302892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:16.304460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:16.314694 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 187/383] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:16.320135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:16.366871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77613ed-54ef-4e55-83ab-db0d82a80fb8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:16.439656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31b7c5-3d2a-4fba-95f4-9481163c26b3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:16.477441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:16.478056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:16.478721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:16.479000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:16.479269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:16.479558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:16.480026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:16.480129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] RBAC: Adding query filter params (group.id=dcca039955b044c4bbee4d33dc591efd, role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=a1dc1eb7c79345caaeb191fa0f9c008f) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:16.480397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:16.480814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:16.481850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:16.482416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:16.484066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:16.486189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 188/384] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:16 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:16.492125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:16.494847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 197/385] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:16 2022] GET /identity/v3/role_assignments?group.id=dcca039955b044c4bbee4d33dc591efd&scope.project.id=a1dc1eb7c79345caaeb191fa0f9c008f&role.id=90dd7541fe26472fb3843454a738e154 => generated 260 bytes in 493 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:16.538711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:16.607477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f3d242-8875-4577-a05e-51685dcb672c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:16.648054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b5324-5cb8-454a-9723-148592b9e856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:16.649377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b5324-5cb8-454a-9723-148592b9e856 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:16.649876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b5324-5cb8-454a-9723-148592b9e856 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:16.650355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b5324-5cb8-454a-9723-148592b9e856 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:16.651920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b5324-5cb8-454a-9723-148592b9e856 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:16.653682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b5324-5cb8-454a-9723-148592b9e856 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:16.656113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 189/386] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:16 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:16.661615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:16.836838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:16.846002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de88fe4d-b758-4155-a8ab-8870a567e29e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:16.888346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0467cf6-7f9b-4d17-98bb-4e20cbe0e305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:16.889244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0467cf6-7f9b-4d17-98bb-4e20cbe0e305 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:16.889526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0467cf6-7f9b-4d17-98bb-4e20cbe0e305 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:16.889798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0467cf6-7f9b-4d17-98bb-4e20cbe0e305 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/groups/ad15286df26446fb824519e151a1feb9/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:16.892039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0467cf6-7f9b-4d17-98bb-4e20cbe0e305 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, group_id=ad15286df26446fb824519e151a1feb9, role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:16.897575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0467cf6-7f9b-4d17-98bb-4e20cbe0e305 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:16.925536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 198/387] 10.222.0.33 () {58 vars in 1442 bytes} [Tue Jul 19 10:51:16 2022] PUT /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/groups/ad15286df26446fb824519e151a1feb9/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:17.089073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844585a-c1a1-4de3-a916-d9b825c2f8f9 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:17.089634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844585a-c1a1-4de3-a916-d9b825c2f8f9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:17.090013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844585a-c1a1-4de3-a916-d9b825c2f8f9 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:17.093223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 190/388] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:17.098509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:17.098800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:17.099070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:17.117945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:17.119152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:17.210533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 199/389] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:17.232885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844585a-c1a1-4de3-a916-d9b825c2f8f9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:17.369869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844585a-c1a1-4de3-a916-d9b825c2f8f9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:17.381107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:17.411387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:17.412187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:17.412465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:17.412756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:17.435978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:17.444504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:17.445530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:17.447024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 191/390] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:17.451834 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:17.585830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:17.611121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c9f081-8bec-4924-b1c5-ae7988b7653f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:17.654753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:17.655860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:17.656355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:17.656862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:17.657823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:17.658331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:17.659926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:17.672528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 200/391] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:17.678172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:17.825358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98ca4b-490d-4879-996a-3855f8829243 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:17.861308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da2dc69-90ce-4a2c-b840-be1aadf0fc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:17.862514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2dc69-90ce-4a2c-b840-be1aadf0fc1b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:17.863008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2dc69-90ce-4a2c-b840-be1aadf0fc1b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:17.863488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2dc69-90ce-4a2c-b840-be1aadf0fc1b None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:17.864904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da2dc69-90ce-4a2c-b840-be1aadf0fc1b None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:17.866544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da2dc69-90ce-4a2c-b840-be1aadf0fc1b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:17.868728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 192/392] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:17 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:17.877771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:18.022854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c599e8-d8d4-4f99-b1e8-1686da753bfb None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:18.065030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745281d-c425-4fb0-8e59-7d1ca664953b None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.065671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745281d-c425-4fb0-8e59-7d1ca664953b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.066159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745281d-c425-4fb0-8e59-7d1ca664953b None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.067991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 193/393] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:18.073690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.073954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:18.074437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.074818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.075104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.075350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.075453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.081752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:18.084205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:18.086448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 201/394] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:17 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:18.092143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:18.096151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:18.098075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:18.125722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:18.142823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:18.219311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 194/395] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:18.269169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be45014-8bac-440e-a208-08728e488b7c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:18.279564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:18.313684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:18.315093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.315529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.315948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/groups/ad15286df26446fb824519e151a1feb9/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.318498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] RBAC: Authorizing `identity:create_grant(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1, group_id=ad15286df26446fb824519e151a1feb9, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:18.320392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:18.336681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 202/396] 10.222.0.33 () {58 vars in 1442 bytes} [Tue Jul 19 10:51:18 2022] PUT /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1/groups/ad15286df26446fb824519e151a1feb9/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 10:51:18.424820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413dacd0-f62e-4d73-a371-d7f473787bc9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:18.463534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:18.464627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.465108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.465527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] PATH_INFO: `/v3/roles/member` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.477557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:18.486667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:18.487819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:18.488842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:18.491240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 195/397] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:18.497110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:18.575936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:18.615842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4d9eea-330e-4c52-9690-c97ef1607606 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:18.654695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:18.656406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.656894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.657313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.658198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:18.658646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:18.660153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b986d7c0-57a1-4270-85e9-7c43754473a0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:18.669372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 203/398] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:18.675618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:18.703075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:18.754556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8453fd-d036-4d33-b4b5-9c509d890d77 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.755414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8453fd-d036-4d33-b4b5-9c509d890d77 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.755828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8453fd-d036-4d33-b4b5-9c509d890d77 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.761707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 204/399] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:18 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:18.774898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.775181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.775521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.794418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71479e5f-459d-43f0-82f1-ebe7bcba730f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:18.805227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:18.809587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:18.831524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:18.832458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:18.832767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:18.833046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:18.840738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:18.840738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:18.840738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 196/400] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:18 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:18.843893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:18.918743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 205/401] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:18.940837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.012824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:19.074813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:19.085976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282bac34-913f-4bc5-bb68-16cd18fae932 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.093885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06aeae5-4deb-4a66-891a-c023beea00d4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.154397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e27eab0-f532-4fa2-93e4-ce4692caa0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.155202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e27eab0-f532-4fa2-93e4-ce4692caa0f0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.155481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e27eab0-f532-4fa2-93e4-ce4692caa0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.155818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e27eab0-f532-4fa2-93e4-ce4692caa0f0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.157555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c40817-7233-4b39-aee2-72a583cac2c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.158971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c40817-7233-4b39-aee2-72a583cac2c7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.159453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c40817-7233-4b39-aee2-72a583cac2c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.159941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c40817-7233-4b39-aee2-72a583cac2c7 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.161457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c40817-7233-4b39-aee2-72a583cac2c7 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:19.163317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c40817-7233-4b39-aee2-72a583cac2c7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:19.165688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 197/402] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:18 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:19.167090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebbe8d3-485e-407f-b658-91c8e40b63c6 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.167630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebbe8d3-485e-407f-b658-91c8e40b63c6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.168110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebbe8d3-485e-407f-b658-91c8e40b63c6 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.169900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 198/403] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:19.171649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebbe8d3-485e-407f-b658-91c8e40b63c6 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.171829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e27eab0-f532-4fa2-93e4-ce4692caa0f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6e27eab0-f532-4fa2-93e4-ce4692caa0f0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:19.180359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:19.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:19.182577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 206/404] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:19.183534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.183534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.183534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.200137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:19.201273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:19.272866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 207/405] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:19.273782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.288250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebbe8d3-485e-407f-b658-91c8e40b63c6 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.330468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:19.336303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.337512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.337995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.338469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964/groups/ad15286df26446fb824519e151a1feb9/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.340756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] RBAC: Authorizing `identity:create_grant(project_id=9320bdfceba6447f80351a3d3e908964, group_id=ad15286df26446fb824519e151a1feb9, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:19.342474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:19.358478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 199/406] 10.222.0.33 () {58 vars in 1442 bytes} [Tue Jul 19 10:51:19 2022] PUT /identity/v3/projects/9320bdfceba6447f80351a3d3e908964/groups/ad15286df26446fb824519e151a1feb9/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:19.360211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.386409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:19.398925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5e0f26-36f9-4f2a-bc22-63e9d3297f30 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.477637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.478228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.478519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.478801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.479554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:19.479858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:19.481307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:19.486397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc693fd-7519-4a3c-b70c-ac198a72a39e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.498709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 208/407] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:19.504897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.526577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.527821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.528383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.528916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.541829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:19.550278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:19.551301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:19.552104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:19.559522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 200/408] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:19.561282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.663471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:19.686330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07383f43-6b2f-40d6-ba4e-8f6cd9709f42 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.709044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed59ec9b-ad2f-4387-95e0-160db2532e53 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.724913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08362943-38b6-4952-92be-5009032ef43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.725584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08362943-38b6-4952-92be-5009032ef43c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.725973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08362943-38b6-4952-92be-5009032ef43c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.726249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08362943-38b6-4952-92be-5009032ef43c None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.727113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08362943-38b6-4952-92be-5009032ef43c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:19.727417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08362943-38b6-4952-92be-5009032ef43c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:19.728869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08362943-38b6-4952-92be-5009032ef43c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:19.729225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:19.741410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 201/409] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:19.742821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323aa1b0-e611-46fe-a8c5-1ed280ac4def None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.743139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323aa1b0-e611-46fe-a8c5-1ed280ac4def None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.743380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323aa1b0-e611-46fe-a8c5-1ed280ac4def None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.744851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 202/410] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:19.750112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323aa1b0-e611-46fe-a8c5-1ed280ac4def None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.754749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62bc1d6-a975-4881-9309-b52c93fe50b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.756108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62bc1d6-a975-4881-9309-b52c93fe50b5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.756548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62bc1d6-a975-4881-9309-b52c93fe50b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.756995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62bc1d6-a975-4881-9309-b52c93fe50b5 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.758322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62bc1d6-a975-4881-9309-b52c93fe50b5 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:19.760408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62bc1d6-a975-4881-9309-b52c93fe50b5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:19.763292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 209/411] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:19.764794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.765260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.765684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.786937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:19.789018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:19.871138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323aa1b0-e611-46fe-a8c5-1ed280ac4def None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:19.884403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 210/412] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:19.885609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.911148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:19.912151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:19.912551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:19.912983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] PATH_INFO: `/v3/groups/dcca039955b044c4bbee4d33dc591efd` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:19.914298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] RBAC: Authorizing `identity:get_group(group_id=dcca039955b044c4bbee4d33dc591efd)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:19.915833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:19.918816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 203/413] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/groups/dcca039955b044c4bbee4d33dc591efd => generated 229 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:19.919632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:19.935342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.022443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc7e4a-e558-4057-bb40-a21fb3c7ca4a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.027714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8f184-8bc0-4ab7-851a-3402fc7bda61 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.040863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.063061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.064006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.064404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.064821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.070147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.071337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.071766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.072432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.074149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.076027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.076157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:20.078847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 211/414] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:20.079857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:20.085323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:20.086265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:20.087131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:20.089171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 204/415] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:20.090110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:20.162665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.209515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adaf80-00f4-4dad-a94a-95294e41a3dc None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.211672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1849d3-d4af-4576-ad6e-72b2a0d2b066 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.246757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.247375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.247645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.247929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.248204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964/groups/ad15286df26446fb824519e151a1feb9/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.248307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.248501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.248810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] PATH_INFO: `/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.250498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] RBAC: Authorizing `identity:get_project(project_id=a1dc1eb7c79345caaeb191fa0f9c008f)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.250985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] RBAC: Authorizing `identity:create_grant(project_id=9320bdfceba6447f80351a3d3e908964, group_id=ad15286df26446fb824519e151a1feb9, role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.251953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.252405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.253972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 212/416] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:20 2022] GET /identity/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f => generated 344 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:20.254738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:20.274340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 205/417] 10.222.0.33 () {58 vars in 1442 bytes} [Tue Jul 19 10:51:20 2022] PUT /identity/v3/projects/9320bdfceba6447f80351a3d3e908964/groups/ad15286df26446fb824519e151a1feb9/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:20.275447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:20.309249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.363737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e34f28-a2ce-4812-b1cb-e8a3288267d7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.398518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec9cbd-c3f0-48d9-ad33-eef400af8784 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.400591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.401241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.401527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.401805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.403278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:20.403278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.404268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.414100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 213/418] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:20.419452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:20.445920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.455328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.456667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.457238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.457728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] PATH_INFO: `/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f/groups/dcca039955b044c4bbee4d33dc591efd/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.460441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] RBAC: Authorizing `identity:create_grant(project_id=a1dc1eb7c79345caaeb191fa0f9c008f, group_id=dcca039955b044c4bbee4d33dc591efd, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.462165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.485150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 206/419] 10.222.0.33 () {58 vars in 1442 bytes} [Tue Jul 19 10:51:20 2022] PUT /identity/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f/groups/dcca039955b044c4bbee4d33dc591efd/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:20.535868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328dbc7c-fd50-436f-8730-7476c9b46db0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.551206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.575316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff094dc-7d81-48dc-b597-8a8edc56cb6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.576030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff094dc-7d81-48dc-b597-8a8edc56cb6c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.576322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff094dc-7d81-48dc-b597-8a8edc56cb6c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.576604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff094dc-7d81-48dc-b597-8a8edc56cb6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.655591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff094dc-7d81-48dc-b597-8a8edc56cb6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.657746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff094dc-7d81-48dc-b597-8a8edc56cb6c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.773400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 214/420] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 19 10:51:20.776624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:20.825866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8d1f2-b6ec-4deb-acdf-7e73d09be3bc None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.826164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8d1f2-b6ec-4deb-acdf-7e73d09be3bc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.826333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8d1f2-b6ec-4deb-acdf-7e73d09be3bc None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.828633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 215/421] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:20.834304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.834656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.834863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.871148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:20.872358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:20.892458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:20.941307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f02d726-79d7-45f4-becf-eee8cb88e9c9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:20.983827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:20.985690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:20.986092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:20.986736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:20.988346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:20.990414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:20.993026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 207/422] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:20 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:20.999149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:21.017953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 216/423] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:21.041180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:21.125138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:21.185448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7d2874-1cf5-41b0-962c-c3e57a0786c2 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:21.201922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f73b8d-3024-4b1f-b6f3-b2368169c8a3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:21.248116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:21.250640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.250938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.251213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.261424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:21.262266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.262543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.262810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.265828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:21.268036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:21.272245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 208/424] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:20 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:21.272804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:21.275956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:21.290257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:21.291162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:21.291929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:21.294265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 217/425] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:21.297483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:21.305015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:21.465673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b60d26-807a-4e87-92ca-88f672dcb4cf None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:21.494793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c2b0f6-5ade-46bc-819d-2f36326079a9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:21.517036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:21.532952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:21.533611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.533897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.534615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.535462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:21.535776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:21.537246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:21.550809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 209/426] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:21 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 611 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:21.555585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:21.556260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.556542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.556866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.557625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:21.557927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:21.559400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3056da4-a450-4782-b42d-7889cfe3af3f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:21.569886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 218/427] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:21.578503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:21.727254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b26766-278b-4dd6-9d7f-5fb7410af515 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:21.764348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:21.765407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.765924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.766407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.834792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:21.849439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:21.851343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:21.868659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc30e32-4ae1-466a-ad97-0b02653e419d None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.869195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc30e32-4ae1-466a-ad97-0b02653e419d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.869271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc30e32-4ae1-466a-ad97-0b02653e419d None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.874691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 219/428] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:21.877384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:21.877677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:21.877955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:21.905390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:21.907922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:21.980074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 210/429] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:21.985736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:21.999446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.013295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 220/430] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:22.047157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:22.093202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.103526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576bfc05-0660-4af9-ad9f-d14ae777ace1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.137735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.138596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.139199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.139199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] PATH_INFO: `/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] RBAC: Authorizing `identity:get_project(project_id=e9122e2aa0cc4d1f8bc0702f21b3a1e1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:22.142504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:22.144617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 211/431] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:21 2022] GET /identity/v3/projects/e9122e2aa0cc4d1f8bc0702f21b3a1e1 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:22.152678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:22.158528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc7d5d6-9d14-4a92-8200-0a9b49a2dcbb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.204335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.205163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.205442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.205817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.217444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:22.226130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:22.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:22.227800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 221/432] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:22.234681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:22.269020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb74f49-c95e-44c2-b117-2f089df2b45c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.306386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.318809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.319678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.319950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.320214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.321431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:22.323801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:22.326247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 212/433] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:22 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:22.333621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:22.362209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.368034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8526897-8767-446c-aa52-aef9940c48d3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.413769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.414439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.414722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.414995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.415742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] RBAC: Adding query filter params (name=member) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:22.416046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:22.417629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:22.426547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 222/434] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:51:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:22.432001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:22.445389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7726db-5bcc-4ef9-ae30-b32eb5ba8408 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.482284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.482948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.483232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.483507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:22.485330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:22.486713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:22.493611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 213/435] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:22 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=e9122e2aa0cc4d1f8bc0702f21b3a1e1&role.id=168377268ae14ad882f881d40dd39b5c => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:22.549376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cdcf50-c63a-4ab5-80d8-03fdfd4e2dc2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.569061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.589772 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a612e93-519f-4329-b6fa-1d5f55999354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.590440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a612e93-519f-4329-b6fa-1d5f55999354 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.590726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a612e93-519f-4329-b6fa-1d5f55999354 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.591004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a612e93-519f-4329-b6fa-1d5f55999354 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.668462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a612e93-519f-4329-b6fa-1d5f55999354 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:22.669957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a612e93-519f-4329-b6fa-1d5f55999354 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:22.775116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 223/436] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 19 10:51:22.780415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:22.876294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0612d5-d4cb-4eb4-99dd-ef88cd0c4dd3 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.876830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0612d5-d4cb-4eb4-99dd-ef88cd0c4dd3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.877121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0612d5-d4cb-4eb4-99dd-ef88cd0c4dd3 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.878882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 224/437] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:22.882851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.885502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.885792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.886226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.891431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec8b06-e0f0-43e3-acdb-f2e90ef2d41e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:22.911864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:22.913113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:22.927106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:22.936196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5f59a0-68a8-4eb7-8b11-5cda8459e860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:22.937463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f59a0-68a8-4eb7-8b11-5cda8459e860 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.937463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f59a0-68a8-4eb7-8b11-5cda8459e860 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.937639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f59a0-68a8-4eb7-8b11-5cda8459e860 None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.938875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5f59a0-68a8-4eb7-8b11-5cda8459e860 None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:22.940664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5f59a0-68a8-4eb7-8b11-5cda8459e860 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:22.943850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 214/438] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:22 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:22.943850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944127fc-adf7-4c92-80b1-966a0797f1fa None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.943850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944127fc-adf7-4c92-80b1-966a0797f1fa None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.944080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944127fc-adf7-4c92-80b1-966a0797f1fa None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.945451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 215/439] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:22.950833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:22.951130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:22.951416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:22.969201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:22.970834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:23.010405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 225/440] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 19 10:51:23.011277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.046962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 216/441] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:23.047738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.129998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff2ef7b-4ff1-4069-9b6a-03d43c901bef None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.142397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.154905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99d6ace-565c-4466-bf24-09e5a69a624c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.163471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.164321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.164601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.164898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.166072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.167514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.169156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 226/442] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:23.169931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.184267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.193561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.194545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.194823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.195091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.205464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:23.213035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:23.213908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:23.214732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 217/443] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:23.215383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.273163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c055e3c-eefb-41fd-b4a1-f656304cf9a0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.305074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.305796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.306075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.306343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.313566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edaaa12-d30f-447a-97b0-67701fe037f5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.316686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:23.324716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:23.325541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 227/444] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:23.327086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.356129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.356803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.357094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.357361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.358192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=168377268ae14ad882f881d40dd39b5c, scope.project.id=9320bdfceba6447f80351a3d3e908964) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:23.358501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.360716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.366830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 218/445] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=168377268ae14ad882f881d40dd39b5c => generated 611 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:23.367429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.394065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.412018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.442635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6642d0-e4a2-445a-9223-bab80d8cb1a1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.481012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.481694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.481977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.482250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.483021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:23.483335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.485571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.494514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 228/446] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:23.500841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.506576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a614bea-56d7-4d6d-ae6d-3fd4b994dc75 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.548646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.549360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.549650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.549925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.550683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:23.551228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.552345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.563277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 219/447] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:23.568848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.613968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90619ed0-e745-4425-afe0-c22486998ae7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.653910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.654563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.654855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.655129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.661865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.664237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.695670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afbea35-ecc2-43bc-9e31-de8b1ceb35e5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.740882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.741534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.741815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.742085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.745036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.746517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.808574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.810050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.851509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 229/448] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 19 10:51:23.857120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.881596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.891302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:23.908306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 220/449] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 19 10:51:23.913783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:23.955106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd108575-b609-464b-a43b-ee0fe123bc98 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:23.988823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:23.989665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:23.989949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:23.990222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] PATH_INFO: `/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:23.991397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] RBAC: Authorizing `identity:get_project(project_id=a1dc1eb7c79345caaeb191fa0f9c008f)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:23.992870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:23.994543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 230/450] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/projects/a1dc1eb7c79345caaeb191fa0f9c008f => generated 344 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:23.999949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:24.014046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec04675-2969-4168-ba1b-b22b64934b3e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:24.047688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:24.048538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:24.048855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:24.049131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] PATH_INFO: `/v3/projects/9320bdfceba6447f80351a3d3e908964` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:24.050316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] RBAC: Authorizing `identity:get_project(project_id=9320bdfceba6447f80351a3d3e908964)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:24.051786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:24.053500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 221/451] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/projects/9320bdfceba6447f80351a3d3e908964 => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:24.059252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:24.100821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:24.108065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f47bf9-58f2-4f4a-93d1-ba4a122af4d1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:24.111587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:24.144061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:24.144923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:24.145207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:24.145478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] PATH_INFO: `/v3/groups/dcca039955b044c4bbee4d33dc591efd` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:24.146639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] RBAC: Authorizing `identity:get_group(group_id=dcca039955b044c4bbee4d33dc591efd)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:24.148061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:24.149687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 231/452] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:23 2022] GET /identity/v3/groups/dcca039955b044c4bbee4d33dc591efd => generated 229 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:24.155322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:24.158214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6486ff9-c4e2-4dcd-942e-c2263364e07f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:24.191617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:24.192460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:24.192775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:24.193054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] PATH_INFO: `/v3/groups/ad15286df26446fb824519e151a1feb9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:24.194234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] RBAC: Authorizing `identity:get_group(group_id=ad15286df26446fb824519e151a1feb9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:24.195657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:24.197317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 222/453] 10.222.0.33 () {56 vars in 1180 bytes} [Tue Jul 19 10:51:24 2022] GET /identity/v3/groups/ad15286df26446fb824519e151a1feb9 => generated 226 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:24.203582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:24.255713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f8e41-e144-4685-83ee-ed7313823616 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:24.290154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:24.290788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:24.291063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:24.291337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:24.292156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] RBAC: Adding query filter params (group.id=dcca039955b044c4bbee4d33dc591efd, role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=a1dc1eb7c79345caaeb191fa0f9c008f) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:24.292457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:24.293905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001a9aab-e30c-4c23-b4ab-46b5a1d30468 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:24.299451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 232/454] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:24 2022] GET /identity/v3/role_assignments?group.id=dcca039955b044c4bbee4d33dc591efd&scope.project.id=a1dc1eb7c79345caaeb191fa0f9c008f&role.id=90dd7541fe26472fb3843454a738e154 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:24.300522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbc89f7-50b2-48ba-a5a4-597321d25460 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:24.312128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:24.338118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:24.338806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:24.339044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:24.339317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:24.340166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] RBAC: Adding query filter params (group.id=ad15286df26446fb824519e151a1feb9, role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=9320bdfceba6447f80351a3d3e908964) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:24.340483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:24.341988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3fde2-af09-486d-8356-8f7ff6e68f55 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:24.349550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 223/455] 10.222.0.33 () {56 vars in 1376 bytes} [Tue Jul 19 10:51:24 2022] GET /identity/v3/role_assignments?group.id=ad15286df26446fb824519e151a1feb9&scope.project.id=9320bdfceba6447f80351a3d3e908964&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:26.925070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df893fcd-08fc-419a-a413-8c76ef42b1a5 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:26.925602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df893fcd-08fc-419a-a413-8c76ef42b1a5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:26.925602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df893fcd-08fc-419a-a413-8c76ef42b1a5 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:26.927202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 233/456] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:26.932656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1625a-a17e-4eca-81d4-e472bc6f1196 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:26.933089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1625a-a17e-4eca-81d4-e472bc6f1196 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:26.933283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1625a-a17e-4eca-81d4-e472bc6f1196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:26.960867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-02c1625a-a17e-4eca-81d4-e472bc6f1196 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:26.962067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-02c1625a-a17e-4eca-81d4-e472bc6f1196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:27.058632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc0ce6e-fb03-40e7-b397-206d5cd9d2e2 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.059304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc0ce6e-fb03-40e7-b397-206d5cd9d2e2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.059562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 224/457] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:27.060006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc0ce6e-fb03-40e7-b397-206d5cd9d2e2 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.061910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 234/458] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:27.067593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.067883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.068161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.093728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:27.094931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:27.122855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc0ce6e-fb03-40e7-b397-206d5cd9d2e2 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.185383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:27.209824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 225/459] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:27.251164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.292806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc0ce6e-fb03-40e7-b397-206d5cd9d2e2 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:27.314642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:27.367150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:27.367956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.368240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.368513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.391781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:27.393282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:27.420629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:27.422078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:27.422914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:27.422914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:27.422914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:27.422914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:27.423135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92604d0a-3624-464d-97a7-79f2a82d3c8d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:27.433029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 235/460] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:27.438262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.476018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:27.476948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.477234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.477507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.489249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:27.490694 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:27.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:27.511123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:27.511962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:27.511962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:27.511962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:27.511962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:27.511962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 226/461] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:27.518700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.592445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:27.599031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac6e738-baa1-4b80-affe-c8a90561bddc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:27.636074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:27.637084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.637549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:27.637667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.638061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.638994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:27.639528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:27.641194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34970711-c58f-40a1-886c-b29d14c33719 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:27.643662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800140d9-c0ab-4f7e-9b6a-f31053186da9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:27.653636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 236/462] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:27.655186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1436721e-85c5-48b6-9043-849e317c2761 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.655680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1436721e-85c5-48b6-9043-849e317c2761 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.656867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1436721e-85c5-48b6-9043-849e317c2761 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.658402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 237/463] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:27.659288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1436721e-85c5-48b6-9043-849e317c2761 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.689669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:27.690248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.690750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.691193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.692202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:27.692810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:27.694364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56bd5d5-dad2-4704-9711-d82a879741e4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:27.713241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 227/464] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:27.714900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d10b55b-a9db-46e9-9a57-d84a6b8976aa None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.715349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d10b55b-a9db-46e9-9a57-d84a6b8976aa None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.715782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d10b55b-a9db-46e9-9a57-d84a6b8976aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.733169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1d10b55b-a9db-46e9-9a57-d84a6b8976aa None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:27.735131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1d10b55b-a9db-46e9-9a57-d84a6b8976aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:27.772083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1436721e-85c5-48b6-9043-849e317c2761 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:27.805417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 228/465] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:27.806900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d32db4-12ec-4962-acb2-f44c1a3f91cc None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.807196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d32db4-12ec-4962-acb2-f44c1a3f91cc None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.807476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d32db4-12ec-4962-acb2-f44c1a3f91cc None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.808759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 229/466] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:27.809646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d32db4-12ec-4962-acb2-f44c1a3f91cc None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.814504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1bb7f9-1d28-4631-b26d-b2ad2238407a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:27.815359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bb7f9-1d28-4631-b26d-b2ad2238407a None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.815791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bb7f9-1d28-4631-b26d-b2ad2238407a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.816231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bb7f9-1d28-4631-b26d-b2ad2238407a None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.817132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bb7f9-1d28-4631-b26d-b2ad2238407a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:27.818767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bb7f9-1d28-4631-b26d-b2ad2238407a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:27.841593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 238/467] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:27 2022] POST /identity/v3/users => generated 246 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:27.843142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:27.844008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:27.844277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:27.851383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:27.864714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:27.867088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:27.938235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:27.960954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 239/468] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:27.967759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:27.972427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d32db4-12ec-4962-acb2-f44c1a3f91cc None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.009336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.010417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.010895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.011361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.013975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.037876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 230/469] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:27 2022] POST /identity/v3/users => generated 248 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:28.039017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.086019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb77e51-cc8a-4dc4-9006-77841ff3510f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.128034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.129123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.129787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.130246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.133747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:28.150767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.153341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:28.180358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:28.181346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:28.182190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:28.182190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:28.182190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:28.182190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:28.182837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 240/470] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:28.188546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.189656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b0a99-a839-4197-af5f-d2d1eab0a49a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.225590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:28.233387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.234800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.235477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.235906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.249540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.251819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:28.272517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:28.273434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:28.274257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:28.274257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:28.274257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:28.274257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:28.275151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 231/471] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:28.281727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.323162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6babf70-823c-4ff8-940d-237bdbb0d408 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.368566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.369279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.369566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.369844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.371309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:28.371309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.372254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0193259f-0ba7-4d55-9f8a-379198498cd3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.385372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 241/472] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:28.386761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88467ace-5693-4526-b210-35295839128b None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.387056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88467ace-5693-4526-b210-35295839128b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.387334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88467ace-5693-4526-b210-35295839128b None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.388566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 242/473] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:28.394592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.394884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.395165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.412722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:28.413928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:28.414849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16df1b2-1c88-46b7-bd1d-5cb166fd3fc8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.452288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.453530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.453866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.454187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.454922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:28.455225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.456720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.458650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:28.473087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 232/474] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:28.474030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.485697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 243/475] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:28.486465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.524960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:28.586317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8889410e-d3c4-4f33-99fd-c39448f29e9e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.624144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d41d6b-f8f3-49a3-82b8-60749666917a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.624583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.625918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.626300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.626694 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.627507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.629048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.652843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 233/476] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:28 2022] POST /identity/v3/users => generated 248 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:28.659024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.663323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.664078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.664366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.664645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.665392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.666879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.706820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 244/477] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:28 2022] POST /identity/v3/users => generated 249 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:28.785445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:28.788649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9d128-753e-4781-8f60-75e91c3dca0c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:28.839544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:28.840250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:28.840534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:28.840843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:28.841628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:28.843082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:28.865312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.sql.core [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 19 10:51:28.865312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 19 10:51:28.865312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [parameters: {'id': 'cb9997e79cf040748904b03a5b52e354', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 19 10:51:28.865312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80152) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result = self._query(query) Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application conn.query(q) Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result.read() Jul 19 10:51:28.882514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return ref.to_dict() Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 19 10:51:28.885008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application next(self.gen) Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application yield resource Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application next(self.gen) Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.session.rollback() Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.force_reraise() Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise self.value Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application session.commit() Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 19 10:51:28.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._prepare_impl() Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.session.flush() Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._flush(objects) Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application compat.raise_( Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application flush_context.execute() Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rec.execute(self) Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 19 10:51:28.889444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 19 10:51:28.896870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result = self._query(query) Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application conn.query(q) Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result.read() Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 19 10:51:28.897754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [parameters: {'id': 'cb9997e79cf040748904b03a5b52e354', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:28.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 19 10:51:28.899322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:28.900133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 234/478] 10.222.0.33 () {60 vars in 1131 bytes} [Tue Jul 19 10:51:28 2022] POST /identity/v3/roles => generated 144 bytes in 231 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:28.900204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:28.942499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:29.019545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4592c0-f524-42e6-b513-0d3ab3424545 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:29.059594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0163f60f-0c43-48c9-a986-f17e73514ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:29.060323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0163f60f-0c43-48c9-a986-f17e73514ece None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:29.060600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0163f60f-0c43-48c9-a986-f17e73514ece None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:29.060902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0163f60f-0c43-48c9-a986-f17e73514ece None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:29.071998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0163f60f-0c43-48c9-a986-f17e73514ece None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0163f60f-0c43-48c9-a986-f17e73514ece None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:29.080581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:29.081530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:29.082333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 245/479] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:51:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:29.087724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:29.199047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ce6cd-9faa-4dc8-bdf1-1a4da8cfb600 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:29.236252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:29.236940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:29.237225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:29.237499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:29.238251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:29.238551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:29.239918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6e873-478e-4005-91d5-853259a4ce52 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:29.248839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 235/480] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:30.461030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fef8a2-b7ce-4915-a816-1d81ae9fdd00 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:30.461574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fef8a2-b7ce-4915-a816-1d81ae9fdd00 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:30.461574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fef8a2-b7ce-4915-a816-1d81ae9fdd00 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:30.463193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 246/481] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:30.468212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:30.469162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:30.469348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:30.472712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a5c9e-c8f7-4529-92b8-5f3c21ada202 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:30.473014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a5c9e-c8f7-4529-92b8-5f3c21ada202 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:30.473297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a5c9e-c8f7-4529-92b8-5f3c21ada202 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:30.475029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 247/482] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:30.479888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:30.480180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:30.480465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:30.493996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:30.495185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:30.510859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:30.512032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:30.528389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:30.612306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 236/483] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:30.633353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 248/484] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:30.646629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:30.659520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:30.711427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:30.814628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421685d0-dde9-43c9-b672-70abc8c79136 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:30.838870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69e7015-f274-4f80-a1d8-93d830af7def None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:30.867814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:30.868610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:30.868912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:30.869185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:30.877929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:30.878672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:30.878947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:30.879213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:30.892123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:30.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:30.900789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:30.901570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:30.905756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:30.914042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:30.914894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:30.915757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 237/485] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:30.916417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 249/486] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:30.925718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:30.926773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:30.970747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.019581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.042692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc94cc-0397-4d28-bf37-a226d25094b0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.105020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.105688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.105969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.106240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.106994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:31.107297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.110726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.128651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 250/487] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:31.134222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.151000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dfe192-334e-41c4-8bf7-431e4b165f05 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.216900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.217569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.217850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.218117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.219431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:31.219733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.221175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.229969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 238/488] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:31.237385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.290371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0342452-32d9-4bd6-901f-8117db0a3796 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.347540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.348337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.348618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.348937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.352511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.361345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.363221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.366577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.375756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc9cffd-88f9-4780-834e-bcb088043b1c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f02cbada-b3d8-4688-9ef0-da3c86aaf256 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:31.386538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:31.387358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:31.388211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:31.388211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:31.388211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:31.388211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:31.388847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 251/489] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:31.391040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0b480-4b06-4755-8cb2-6e128bf0dfdc None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.391325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0b480-4b06-4755-8cb2-6e128bf0dfdc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.391598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0b480-4b06-4755-8cb2-6e128bf0dfdc None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.393719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 252/490] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:31.398643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d0b480-4b06-4755-8cb2-6e128bf0dfdc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.411362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.412153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.412438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.412734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.424261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.425740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2e2b5b21-9b40-465b-9f90-9ac307c0c7a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:31.443414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:31.444274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:31.445303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:31.445303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:31.445303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:31.445303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:31.445303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 239/491] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:31.445831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.446114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.446387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.468420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:31.469672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:31.525235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d0b480-4b06-4755-8cb2-6e128bf0dfdc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.548775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 240/492] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:31.548775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.567337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.567992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.568429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.568722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.569449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:31.569948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.571181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180f9d8e-947c-4b10-a10b-7353eb4fce9a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.584311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 253/493] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:31.585879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a50d0-6bfe-4fe8-8c3a-c31d25268e37 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.586340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a50d0-6bfe-4fe8-8c3a-c31d25268e37 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.586793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a50d0-6bfe-4fe8-8c3a-c31d25268e37 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.588159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 254/494] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:31.589019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12a50d0-6bfe-4fe8-8c3a-c31d25268e37 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.626018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.649623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.673405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50309052-b1f0-4b5c-ab6c-ba7fde2f7291 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.700197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12a50d0-6bfe-4fe8-8c3a-c31d25268e37 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.709864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.710650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.710937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.711214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.711942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:31.712249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.713779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.731075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 241/495] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:31.731075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.742947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8593fac6-a5b0-43b4-bc32-2b8ff61247e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.743738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8593fac6-a5b0-43b4-bc32-2b8ff61247e1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.744016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8593fac6-a5b0-43b4-bc32-2b8ff61247e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.744291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8593fac6-a5b0-43b4-bc32-2b8ff61247e1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.754868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8593fac6-a5b0-43b4-bc32-2b8ff61247e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-8593fac6-a5b0-43b4-bc32-2b8ff61247e1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:31.762500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:31.763566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:31.764372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 255/496] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:31.765047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.765127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.765426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.781563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:31.782664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:31.860730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 256/497] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:31.860730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.861084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6e399a-ab22-465d-8228-992b582b9fa3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:31.883549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.899883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:31.900806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:31.901160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:31.901594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] PATH_INFO: `/v3/users/glance` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:31.906154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:31.910851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:31.912284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:31.936577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:31.938336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 242/498] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/users/glance => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:31.939260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:31.966060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1628f7-4f01-4ce3-98dc-6c5d6f07b92b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.002061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.002844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.003124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.003393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.010388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.011827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:32.027442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:32.028337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:32.029250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 257/499] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:32.029829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.049447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d0e3f-97f7-436b-bef3-8cbd16e9a69c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.082748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.083407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.083686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.083958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.084728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:32.085041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.086408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.094747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 243/500] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:32.095515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.132549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51cffe5-2787-429e-84a4-823189f35c84 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.136977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.140499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.167895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ad0a7c-4f63-4a9a-9c96-e449d0ab0f24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.168670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad0a7c-4f63-4a9a-9c96-e449d0ab0f24 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.168985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad0a7c-4f63-4a9a-9c96-e449d0ab0f24 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.169256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad0a7c-4f63-4a9a-9c96-e449d0ab0f24 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.179762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ad0a7c-4f63-4a9a-9c96-e449d0ab0f24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-51ad0a7c-4f63-4a9a-9c96-e449d0ab0f24 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:32.187467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:32.188320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:32.189422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 258/501] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:32.190326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3608bbbd-d57b-4592-b111-1e098be25442 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.190393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3608bbbd-d57b-4592-b111-1e098be25442 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.190665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3608bbbd-d57b-4592-b111-1e098be25442 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.191910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 259/502] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:32.192589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3608bbbd-d57b-4592-b111-1e098be25442 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.200965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e17df7e-c52a-41cd-ba87-12d874cf7634 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.235808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.236463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.236776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.237051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.237879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:32.238190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.239602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.274535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 244/503] 10.222.0.33 () {56 vars in 1137 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:32.275356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.309588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3608bbbd-d57b-4592-b111-1e098be25442 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.341865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.342528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.342813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.343090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.344412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:32.344744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.346175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.372120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.375353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aba3d26-fddf-4849-bbb8-2139e267a5f2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.382519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 260/504] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:32.383260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.408296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.411660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.412045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.412337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.412610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.422722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.424177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-c1c712d0-4d7e-430a-b24d-4874283655da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:32.441264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:32.442248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:32.443251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:32.443251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:32.443251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:32.443251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:32.443251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 245/505] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:32.443642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.443869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.444144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.460210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:32.461451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:32.484507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae05d8-916a-4403-9b22-26faaed7d8eb None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.518296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.518956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.519247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.519525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.520278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:32.520591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.522372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.526373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 246/506] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:32.527051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.535257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 261/507] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:32.536072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.627912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.637825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.644408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9bd1dd-4eb1-43b2-b295-30c42f8d4416 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.657743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4259f700-05e0-457d-aec6-2391c378b251 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.677177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.677808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.678088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.678359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.691646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.692285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.692566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.692879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.743220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.744668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.761921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.763386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:32.826010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.842227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:32.843957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 262/508] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:32.844573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.864624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 247/509] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:32.868879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:32.962991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28a57ed-6c46-44d4-828b-037ce239799a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.974137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250d75f2-6386-41e2-b94d-f9bc84077d2b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:32.995879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:32.996530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:32.996851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:32.997130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:32.997845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:32.998150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:32.999532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.009038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.009812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.010091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.010358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.011210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 263/510] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:33.011824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.023191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.024636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:33.042911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:33.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:33.049719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:33.049719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:33.049719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.049719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:33.049719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 248/511] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:33.049719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.056460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.079893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.111333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881cab6c-87f6-40f6-a150-4f0152756117 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.143777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.144379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e0a88-a1c6-40b1-884e-f973816ece15 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.144557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.144870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.145150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.155756 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.157277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:33.174351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:33.175317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:33.176314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:33.176314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:33.176314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.176314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:33.176314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 264/512] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:33.176314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.179704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.180557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.180922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.181197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.191757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.193170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:33.210046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:33.211028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:33.211876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:33.211876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:33.211876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.211876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:33.211876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 249/513] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:33.212143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.274264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.277787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f815446-3d95-4512-a5c1-1753ffd9b8a6 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.297393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.310582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.311362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.311640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.311907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.312025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dc9e31-1541-4b0d-b301-711614c16743 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.323083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.324543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:33.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:33.342093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:33.343132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:33.343132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:33.343132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:33.343132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:33.345180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.345958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.346237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.346508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.347033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 265/514] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:33.347716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.353616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.355028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:33.369979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:33.371029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 250/515] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/users/neutron => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:33.371936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.447570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bb67a-e327-4c61-ae03-96908b2815e9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.471436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1bbda-c287-4204-84be-81849781c826 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.480226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.480893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.481175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.481446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.482154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:33.482451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.483818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.495932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 266/516] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:33.496733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.507769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.511161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.511819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.512106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.512387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.513239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:33.513548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.514906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.526357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 251/517] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:33.527131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.535889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.599340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61de6c36-4b5f-48e4-8e0d-03a4836262f7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.624355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f43ba-b10b-48db-87fe-25eb048a954a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.634533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.635183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.635470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.635747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.636465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:33.636806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.638185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.649296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 267/518] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:33.650433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.661993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.662649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.662936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.663211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.663933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:33.664239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.665624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.678210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 252/519] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:33.679000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.744538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.748664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8cbfd4-5f42-4c6c-96a7-eb3368a9e652 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.779634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1f0613-d29d-437a-abd2-3e853b7ad3ad None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.781886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.782545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.782829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.783101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.783921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:33.784226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.785663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.802642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 268/520] 10.222.0.33 () {56 vars in 1139 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:33.803404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.817911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.818697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.818980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.819252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.826328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.827793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:33.842079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:33.843109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:33.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 253/521] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/users/cinder => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:33.845162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.899362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8498a5-50b6-4528-aca9-8cd96fce32bc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.926757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.936729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.937426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.937707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.937981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.938677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.940127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:33.945427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f067115-59d5-4071-818f-5b4dd9218921 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:33.961308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 269/522] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:33 2022] POST /identity/v3/users => generated 247 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:33.962370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:33.973913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:33.983487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:33.984396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:33.984678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:33.984979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:33.993087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:33.994572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:34.010069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:34.010990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:34.011825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 254/523] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:34.012429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.107653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe6afff-7fb1-4f46-a43d-0ae650ae7855 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.127500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a590376-1aa3-4848-8e15-a5ac7cd0631e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.143082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.143880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.144170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.144448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.154011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.155472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.167660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.168528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.168848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.169128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:34.173037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:34.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:34.174710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 270/524] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:34.175173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.223827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.234674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.250737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.252194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.286053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5910dc-fc8b-4fdb-a619-e89f78e3e973 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.321106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.321862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.322142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.322420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.323249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:34.323554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.325006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.346804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 271/525] 10.222.0.33 () {56 vars in 1137 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:34.346804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.371612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 255/526] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:34.372494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.447168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d1d10a-2aa7-41e1-b8c9-843320899c24 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.450114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.463897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.479940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e0ec4f-a536-4c84-a26d-6ac55d67f9ef None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.486098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.486705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.486992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.487366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.488066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:34.488374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.489848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.508159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 272/527] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:34.509043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.524936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.525882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.526337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.526791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.527586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:34.528142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.529359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.558370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 256/528] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:34.558370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.622822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a122573-aef9-420b-8e47-fdc61c4f2386 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.661532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.662184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.662481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.662746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.668952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15def4b9-8d55-4e2b-8126-894f7fc8e78d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.695314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.709905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.711007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.711294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.711571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.722382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.723837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.731498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.737815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.739218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:34.743979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:34.744861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:34.745746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:34.745746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:34.745746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:34.745746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:34.752989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 257/529] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:34.753632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.846585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 273/530] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:34.847754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.867778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296ace17-58c5-4538-9178-f79aa1865b39 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:34.912879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:34.913900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:34.914408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:34.914884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:34.916407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2a1c13591bab47d7ba589e14b83f969e) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:34.916963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:34.918612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:34.927207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:34.927490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 258/531] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/role_assignments?user.id=2a1c13591bab47d7ba589e14b83f969e&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:34.928606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:34.991367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaf0b59-59fe-4d90-a690-55bdd22db8f2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.007542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:35.029829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.030493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.030786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.031578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.032434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=9f86b27721074978b12f34dd591362ce) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:35.032768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.034244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.040504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 274/532] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/role_assignments?user.id=9f86b27721074978b12f34dd591362ce&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:35.041384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.057650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8241a6ef-f8e8-449f-8ceb-d0a1e6f8cbf9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.104492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.105256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.105517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.105795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.112098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:35.112098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.112511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.132797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 259/533] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:35.138646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.193357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da13b5c-0e77-4e17-aa90-dbcb2609e228 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.225182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:35.234841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.235993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.236276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.236552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.250152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.251535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.263795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2103362f-ee5d-432e-9f05-559fd3a5220e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:35.271272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:35.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:35.272985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:35.272985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:35.272985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:35.272985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:35.272985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 275/534] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:35.278112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.299267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:35.316654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.317503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.318211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.318494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.327900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.329377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:35.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:35.347915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:35.348818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:35.364544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 260/535] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:35.365483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.432724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3fb405-446d-4827-85d8-07f723d5e7b8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.475656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.476323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.476674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.476910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.477642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:35.477947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.479316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.500750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 276/536] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:35.508585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb58f0-d55d-4635-a425-26b47c047369 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.509690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.554012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.554684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.554966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.555237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.556031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:35.556339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.558560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.576462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 261/537] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:35.582481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.584559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:35.631567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:35.648204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88db8428-d472-4ce0-8536-3b88635ff457 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.683511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.684535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.685002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.685454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.691952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db68b1d7-7698-41f0-b493-2f966eb37e03 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.692944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.694581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:35.709246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:35.710175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:35.710998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:35.713504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 277/538] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:35.725656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:35.744733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.744733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.744733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.744733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.745132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=ea109e86f77f4660b6538e0ec4676629) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:35.745132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.745702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.757859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 262/539] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/role_assignments?user.id=ea109e86f77f4660b6538e0ec4676629&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:35.845022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:35.849298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e10576d-9787-48e9-9b2f-416a631b16e5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:35.886540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:35.887256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:35.887540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:35.887813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:35.888621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:35.888962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:35.891065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389feba1-fe9b-42ea-a0de-72a279d1c549 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:35.908033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 278/540] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:35.913959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:36.057360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb18ea2-9bb0-440e-8238-8cb2bf5e789d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:36.072305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:36.107752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:36.108420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:36.108719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:36.109006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:36.110382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=102f9668a5f948f99ac4ba6fc639c8ad) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:36.110959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:36.112408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b236b030-528a-44f2-a969-8fd901574fa6 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:36.119384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 263/541] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:35 2022] GET /identity/v3/role_assignments?user.id=102f9668a5f948f99ac4ba6fc639c8ad&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:36.149632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd9e96-82a0-4add-9dff-3ceffd6cc117 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:36.149969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd9e96-82a0-4add-9dff-3ceffd6cc117 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:36.150200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd9e96-82a0-4add-9dff-3ceffd6cc117 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:36.151959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 279/542] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:36.157950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:36.158245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:36.158524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:36.177389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:36.178701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:36.266298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 264/543] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:36.304562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd9e96-82a0-4add-9dff-3ceffd6cc117 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:36.450362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd9e96-82a0-4add-9dff-3ceffd6cc117 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:36.497862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:36.498663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:36.498942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:36.499216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:36.512621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:36.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:36.521186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:36.522082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:36.523200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 280/544] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:36.528855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:36.643663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:36.646994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4dac5-8856-4711-bfcf-1a9ab1f8fcdb None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:36.687286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:36.687955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:36.688239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:36.688509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:36.689281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:36.689582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:36.690933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:36.703422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 265/545] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:36.708845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:36.742134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:36.856456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93559f-3002-464b-98c2-14ca666928dc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:36.891850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:36.892931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:36.893216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:36.893490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:36.904183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:36.905675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:36.942313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:36.943129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:36.944397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:36.944397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:36.944397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:36.944397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:36.944397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 281/546] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:36.949497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:37.056929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ece7a5-31e5-4d04-adab-03c3b68daead None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:37.092627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:37.093375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.094520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.094803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.095532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:37.095838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:37.097350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:37.111374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 266/547] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:37.116814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:37.208866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:37.248568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4584cf-234b-477d-9546-9ea185173e21 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:37.296035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:37.297214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.297502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.297780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.308431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:37.309846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:37.326393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:37.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:37.329798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 282/548] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:37.333879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:37.368272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:37.451953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3704fe-5fd5-472b-9f49-176f7a916f92 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:37.488583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:37.489311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.489595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.489869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.490727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:37.491038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:37.492497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ad386-9739-4f9d-b6de-4ccf0b1c9654 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:37.519432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 267/549] 10.222.0.33 () {56 vars in 1135 bytes} [Tue Jul 19 10:51:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:37.524565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:37.567572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664c2e2-31e2-432c-8d28-fd9fc4138fc4 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.567854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664c2e2-31e2-432c-8d28-fd9fc4138fc4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.568042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7664c2e2-31e2-432c-8d28-fd9fc4138fc4 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.569621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 268/550] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:37.575299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.576027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.576531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.595513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:37.597235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:37.673916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1af169-a3ba-4a3d-9b79-d0a9a0590a77 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:37.731119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:37.744719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 269/551] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:37.745176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7558e-a014-414c-aa6b-6a5715fcdd15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:37.745176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7558e-a014-414c-aa6b-6a5715fcdd15 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.745176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7558e-a014-414c-aa6b-6a5715fcdd15 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.745176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7558e-a014-414c-aa6b-6a5715fcdd15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.799668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:37.819120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:37.826515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7558e-a014-414c-aa6b-6a5715fcdd15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:37.828650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7558e-a014-414c-aa6b-6a5715fcdd15 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:37.928725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f294bcfe-2c52-4454-9fb3-3ab40d5fe2d1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:37.933452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 283/552] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:37.940814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eee815-89f6-4953-9b4e-fc4a48afed21 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.940814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eee815-89f6-4953-9b4e-fc4a48afed21 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.940814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eee815-89f6-4953-9b4e-fc4a48afed21 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.940814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 284/553] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:37.942019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eee815-89f6-4953-9b4e-fc4a48afed21 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:37.963016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a7297-f86e-4223-8a5d-a20b4fa741d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:37.963822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a7297-f86e-4223-8a5d-a20b4fa741d5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.964106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a7297-f86e-4223-8a5d-a20b4fa741d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.964441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a7297-f86e-4223-8a5d-a20b4fa741d5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:37.974145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:37.977276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a7297-f86e-4223-8a5d-a20b4fa741d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-c63a7297-f86e-4223-8a5d-a20b4fa741d5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:37.985381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:37.986767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:37.987569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:37.991236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 270/554] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:37.997564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:37.998049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:37.998525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.017901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:38.019385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:38.049570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eee815-89f6-4953-9b4e-fc4a48afed21 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.086688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.087394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.087821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.088037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.089123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.096808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 271/555] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:38.096808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.107947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.109605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:38.146061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:38.147098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:38.147957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:38.147957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:38.147957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:38.147957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:38.147957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 285/556] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:38.148410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.208205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7960d3bc-7fe9-41ab-95ae-8a9b410083a7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.234863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.247869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.248966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.249400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.250342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.251266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:38.251782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.253431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.255121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66152ac8-d124-4227-8bee-c45537b1b442 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.267827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 272/557] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:38.268955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.291144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.291941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.292223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.292493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.303490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:38.311538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:38.313651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:38.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:38.316943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 286/558] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:38.317717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.341217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.385393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d95f39-c1c0-40fb-88cc-f8f4978f3140 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.422953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.423872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.424393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.424952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.425845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:38.426300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.427751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a9b1c-e6f1-4f69-81e7-4265e7d687a4 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.428026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.441991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 273/559] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:38.443319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.463777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.464566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.464892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.465164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.476166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.477640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.479412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:38.497180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:38.498168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:38.499042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:38.499042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:38.499042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:38.499042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:38.499042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 287/560] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:38.499325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.555275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20997821-b1bf-48c7-8d41-71523afc8384 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.579712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.593671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.594373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.594651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.594920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.595667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:38.595969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.597733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e622ec-8ff4-472c-88f1-fc6f16602884 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.612079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 274/561] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:38.612079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eace0195-b9e5-446b-89b2-49910a5fe5ec None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.612079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eace0195-b9e5-446b-89b2-49910a5fe5ec None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.612079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eace0195-b9e5-446b-89b2-49910a5fe5ec None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.612079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 275/562] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:38.612079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eace0195-b9e5-446b-89b2-49910a5fe5ec None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.619720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ee863a-3dd3-499f-8b36-4a3f7aeaa881 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.655033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.656083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.656363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.656672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.663371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.664842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:38.679213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:38.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:38.681063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 288/563] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:38.681988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.716563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eace0195-b9e5-446b-89b2-49910a5fe5ec None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.737183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.750094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.750747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.751036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.751307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.752023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:38.752327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.753700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5fb1a-b8eb-4076-b642-97fb33db9f39 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.765541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 276/564] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:38.766768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d9766-e6d7-4492-a2cf-fa0585d1f14c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.767058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d9766-e6d7-4492-a2cf-fa0585d1f14c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.767333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d9766-e6d7-4492-a2cf-fa0585d1f14c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.783162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c37d9766-e6d7-4492-a2cf-fa0585d1f14c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:38.784336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c37d9766-e6d7-4492-a2cf-fa0585d1f14c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:38.785458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcbce77-4eaa-4525-873e-b0ab7186fce4 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.821320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.822097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.822377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.822646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.826013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.832605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:38.834067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:38.851783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:38.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:38.853967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:38.853967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:38.853967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:38.853967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:38.854241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 277/565] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:38.854877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fab102-c894-46e9-9ec4-34c606bfb8a6 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.855167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fab102-c894-46e9-9ec4-34c606bfb8a6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.855442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fab102-c894-46e9-9ec4-34c606bfb8a6 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:38.861720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 278/566] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:38.861883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 289/567] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:38.861978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fab102-c894-46e9-9ec4-34c606bfb8a6 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.862074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:38.961983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57da9099-120f-4fa5-a00d-99a1c3cae572 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.969014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:38.981905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fab102-c894-46e9-9ec4-34c606bfb8a6 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:38.995978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:38.996785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:38.997069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:38.997341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.004881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.006397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.016423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.017102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.017391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.017667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.018450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.018769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.020189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a05a42-11a5-426a-be5d-ce83e5a2afc1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:39.020876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:39.021891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:39.023014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 290/568] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:39.023706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.033994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 279/569] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:39.035291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.035701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.035975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.052418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:39.053725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:39.080400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.119696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdda0f4-fa86-4ab4-9e25-6872a059d3d4 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.121098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 280/570] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:39.121913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.154841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.155501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.155786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.160100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.160885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.161199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.162574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.174423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 291/571] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:39.175238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.206854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.224390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34751717-9dfa-49b8-a414-f47cb64a393b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.258552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.259333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.259613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.259884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.273458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:39.281126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:39.282174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:39.283128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 281/572] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:39.283945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.291456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56b65a-9842-4595-aa00-79c853b17ce1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.309813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.326429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.327111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.327389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.327658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.328533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.328870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.330276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.346658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 292/573] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:39.347443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.384994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab64239-1fe3-46c5-b0fc-6117025fb374 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.417840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.418484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.418770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.419046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.419898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=110d74df72174fba99493dedd2e98d08) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.420213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.422181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.427516 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 282/574] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/role_assignments?user.id=110d74df72174fba99493dedd2e98d08&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:39.428281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.429933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.448231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8fb517-d95f-4921-9998-9371e4c49e4e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.488056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.488888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.489173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.489451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.502304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:39.510279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:39.511448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 293/575] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:39.517776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.543620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.560264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305f10e6-e3c2-45b5-b524-44b9b37b0e1a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.604477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.605326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.605613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.605887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] PATH_INFO: `/v3/users/glance` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.614530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.615986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:39.631368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:39.632319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 283/576] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/users/glance => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:39.633881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.637679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d405047b-ec1a-422e-ba8a-590d39485034 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.676129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.676985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.677269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.677691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.678446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.678811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.680191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.693527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 294/577] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:39.694262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.703992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.744243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01164415-c7fd-4ddb-89c2-4db32cbe30fb None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.776909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.777702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.777979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.778254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.789194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.790621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.799021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.804052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd85e6-aae0-49c4-8540-a53c41b51d1e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:39.809430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:39.810440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:39.811342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:39.811342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:39.811342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:39.811342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:39.811342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 284/578] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:39.811618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.842902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.844079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.844358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.844631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.845395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.845695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.847073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:39.855187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 295/579] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:39.856013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:39.925609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:39.939448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1972dd-70a0-4d36-8e74-e3920a38fbc0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.965705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3614f-f835-4fdd-b47f-f35a04bb7dfd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:39.978310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:39.979106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:39.979488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:39.979865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:39.980830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:39.981277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:39.982849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.001634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.001898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 285/580] 10.222.0.33 () {56 vars in 1137 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:40.002640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.002980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.003112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.003455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.017387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.018936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:40.036777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:40.037877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:40.039193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:40.039193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:40.039193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:40.039193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:40.039193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 296/581] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:40.043203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.073100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.117963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfa73c-f8fe-4953-95df-376ec389bf75 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.146421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2845fc-ac62-4ad9-9662-046d38f1fee1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.157679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.158452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.158834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.159202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.160027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:40.160748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.161900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.167885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.174928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 286/582] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:40.175573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.188614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.189382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.189675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.189938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.201330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.202753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:40.222798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:40.223785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:40.224846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:40.224846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:40.224846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:40.224846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:40.230538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 297/583] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:40.230538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.284168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0f309-c317-430c-b41d-c649cfdf073a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.301332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.320360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.321205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.321461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.321734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.333471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.334969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.337698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ab944-b3de-45bd-b494-189979ec37c9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:40.351801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:40.352863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:40.358681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:40.358681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:40.358681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:40.358681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:40.358681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 287/584] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:40.358681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.379400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.380176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.380554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.380970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.381793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:40.382191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.383642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.396461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 298/585] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:40.397453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.430075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.473014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac454a-f800-428a-918f-0e1bf5954cc0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.503739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2894a3-e9c6-4477-86ed-9113e4567e9a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.509519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.510491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.510875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.511248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.519750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.521426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:40.538060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:40.539112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:40.540152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:40.541917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 288/586] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:40.542827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.548574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.549267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.549546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.549820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.550545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:40.550847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.552235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.568534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 299/587] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:40.568534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.589831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.660844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.670842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c13167-754c-4007-a328-9bc7feeda9a5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.671008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9eb693-61d9-4bc8-a0ac-e21b95dd06c8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.706196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.706985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.707266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.707532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.708623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.709309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.709601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.709867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.710590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:40.710894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.712247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.714958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.716377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.728874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 289/588] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:40.729525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:40.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:40.731560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:51:40.732577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:40.736827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 300/589] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/users/cinder => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:40.737844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.825199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:40.835673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd3d818-7efa-45f8-a7ed-d99c42e22887 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.842491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062c57cb-49d4-4c5a-8d99-d1f607c9024a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:40.870112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.871012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.871297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.871571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.872355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:40.872665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.874115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.881227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:40.882330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:40.882810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:40.883260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:40.886701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 290/590] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:40.887310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.892133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:40.893812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:40.908432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:40.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 19 10:51:40.910628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:40.912509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 301/591] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/users/neutron => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:40.913582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:40.930260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.017188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893303ee-ee14-49b0-9a5d-5726f0b3a2a2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.019476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f91be-39e2-4a64-8c19-b0e18d2d20fe None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.053157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.053471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.053947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.054113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.054207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.054359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.054511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.054671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.055472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:41.055786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.057247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.061462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.063410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.080584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 302/592] 10.222.0.33 () {56 vars in 1137 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:41.080584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:41.081254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:41.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 291/593] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:41.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.094604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.157016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.190205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ce8f75-d69e-43e4-b763-2cb5152ac33e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.190992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95457663-1f70-4d46-ad35-9b206d399cf1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.227836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.228031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.228504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.228773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.228872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.229058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.229153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.229330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2a1c13591bab47d7ba589e14b83f969e/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.229974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:41.230283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.231834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.244751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=2a1c13591bab47d7ba589e14b83f969e, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.246179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.251750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 292/594] 10.222.0.33 () {56 vars in 1139 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:41.252668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.266348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 303/595] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:41 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2a1c13591bab47d7ba589e14b83f969e/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:41.267526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.346026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.386284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e63f6a-0385-4d92-ad8d-29de38c3fa66 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.406620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246b2f32-9b55-445a-b94d-5998e69e2555 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.430348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.431258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.431540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.431809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.434201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.447558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.449653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.455788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.456651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.457545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.457971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.459308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:41.460120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.461729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cca94c-2a58-4630-83e6-6495e97487e3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:41.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:41.469093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:41.470257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:41.470257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:41.470257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:41.470257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:41.470257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 293/596] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:41.470257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.477244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 304/597] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:41.478876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c4394-42c7-475a-afee-74b652894463 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.479059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c4394-42c7-475a-afee-74b652894463 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.479427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c4394-42c7-475a-afee-74b652894463 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.484836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 305/598] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:41 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:41.485705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412c4394-42c7-475a-afee-74b652894463 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.588869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412c4394-42c7-475a-afee-74b652894463 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.592943 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db29e8-3dc5-4f93-a3c5-8fdb7c379d25 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.613551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.626905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.627592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.627845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.628139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.628859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:41.629187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.630632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.630748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.631415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.631965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.642775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.644215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.647759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 306/599] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:41.648415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-cd77e8d1-7070-49a6-9e06-2f13e1b5477c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:41.661925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:41.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:41.664294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:41.664294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:41.664294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:41.664294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:41.664294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 294/600] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:41.664580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.664868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.665134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.681464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:41.682657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:41.697810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.755609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 295/601] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:41.756505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.771694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261b1f79-f6e4-4332-9ba4-45435d10d788 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.811584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.812533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.812860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.813136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/9f86b27721074978b12f34dd591362ce/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.827954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=9f86b27721074978b12f34dd591362ce, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.829415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.852833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 307/602] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:41 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/9f86b27721074978b12f34dd591362ce/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:41.852833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.874297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce75323-a412-498e-a5a8-fef218e68529 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:41.876419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.928544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:41.931975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:41.931975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:41.931975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:41.943635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:41.945442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:41.964196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:41.967720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:41.970166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:41.971077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:41.974057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 296/603] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:41.975275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:41.982630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f22b3b-83d4-49ab-a62e-ac0828b842c5 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.034382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.035226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.035612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.035991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.036885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:42.037300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.038774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.058929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 308/604] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:42.059785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.100174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad61e319-c1d0-492a-859c-72943bcab514 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.136431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.137187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.137460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.137739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.149951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:42.158368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:42.159297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:42.160124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 297/605] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:42.160846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.185274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40ce4c-e546-4fe8-9565-cd8b04197628 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.192718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:42.234417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.235649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.236367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.236863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.238115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:42.238609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.241155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.261769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 309/606] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:42.261769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.275416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:42.293441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca335910-b036-4e9b-bf7d-6f7e15d1fede None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.341789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.342910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.343345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.344912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.355399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.357094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.371183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3804f36-59a5-4476-a179-af19b8d2251a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:42.377542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:42.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:42.379675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:42.382724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 298/607] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:42.383797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.410902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.411508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.411804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.412064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.412917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:42.413231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.414657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.424150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 310/608] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:42.425095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.482072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:42.497666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:42.504967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b511b-03b3-4bc8-904d-0732d45308f8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.529222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3070f665-4fee-41c8-82d2-fa2097d0b803 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.543342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.544439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.544903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.545416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/102f9668a5f948f99ac4ba6fc639c8ad/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.562178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=102f9668a5f948f99ac4ba6fc639c8ad, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.563612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.564269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.564555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.564860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.565136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.565648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:42.565956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.567364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.590108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 311/610] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:42.590108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.590398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 299/610] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:42 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/102f9668a5f948f99ac4ba6fc639c8ad/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:42.590398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:42.772636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3734265f-3062-458f-96fb-70c6dcac8e1e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.797120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790030e9-a4cb-4e1e-a96e-451be9b76c20 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:42.818335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:42.823699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.826224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.826224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.826224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/ea109e86f77f4660b6538e0ec4676629/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.831843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:42.841629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:42.842656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:42.843134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:42.843532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:42.844520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=ea109e86f77f4660b6538e0ec4676629, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.846022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.855585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:42.857115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:42.865011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 300/611] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:42 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/ea109e86f77f4660b6538e0ec4676629/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:42.882977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:42.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:42.884870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:42.884870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:42.884870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:42.884870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:42.884870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 312/612] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:42.893955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:43.051314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b12d4e-2833-4729-b1f4-1771389c0c6a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:43.094685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02775375-da20-485d-a549-c8d28a5717be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:43.095400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02775375-da20-485d-a549-c8d28a5717be None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.095688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02775375-da20-485d-a549-c8d28a5717be None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.095966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02775375-da20-485d-a549-c8d28a5717be None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.096750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02775375-da20-485d-a549-c8d28a5717be None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:43.097071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02775375-da20-485d-a549-c8d28a5717be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:43.098498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02775375-da20-485d-a549-c8d28a5717be None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:43.113057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 301/613] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:43.118583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:43.261367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ac526-6807-4d4f-af43-f16ac64dbdde None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:43.308769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:43.309493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.309778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.310054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.321767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:43.323241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:43.345689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:43.346653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:43.347562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 313/614] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:43 2022] GET /identity/v3/users/swift => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:43.357836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02775375-da20-485d-a549-c8d28a5717be None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:43.433065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:43.510355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02775375-da20-485d-a549-c8d28a5717be None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:43.557499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:43.558167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.558448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.558729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.559637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:43.559900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:43.561360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1180da-9e0c-4b7a-a325-4ce437ef6e54 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:43.586545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 302/615] 10.222.0.33 () {56 vars in 1135 bytes} [Tue Jul 19 10:51:43 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:43.592302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:43.608158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cf9ab-36c3-4115-b5e7-445d8f4fa3c4 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.608439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cf9ab-36c3-4115-b5e7-445d8f4fa3c4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.608749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147cf9ab-36c3-4115-b5e7-445d8f4fa3c4 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.613045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 303/616] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:43 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:43.619350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.619640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.619919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.653442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:43.654648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:43.675601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:43.737283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335b9f9-adfc-47a4-83e4-e778bb853592 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:43.774339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:43.775126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.775404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.775675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.786783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:43.788241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:43.805863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:43.806760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:43.807649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:43.807649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:43.807649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:43.807649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:43.807649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 314/617] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:43.816160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:43.838384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 304/618] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:43.879855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:43.914956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:43.927060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987972a3-3ac8-43b5-858f-d2598f637bdf None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:43.944084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:43.973260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:43.974370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:43.974850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:43.975319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:43.976257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:43.976837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:43.978406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:43.991412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 315/619] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:44.001240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.080030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3637f9-2639-4400-bf99-28f5dbd1d0a9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.154599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.155429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.155715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.156029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.164033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda4f9da-d3ba-4d57-918d-fff40c07f3e0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.181700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:44.192719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:44.193777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:44.194606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 305/620] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:44.199642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.228239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.229069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.229351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.229619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.244200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:44.245849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:44.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:44.270556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:44.271442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:44.272311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 316/621] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:44.280026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.303992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:44.352717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:44.365402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995511e-5a3c-4b49-be1f-c16a1e1078a2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.403179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c6056d-a27c-4f13-aaf4-9d05a754ec57 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.408219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.409149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.409619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.410075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.410999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:44.411874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:44.413812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc0a549-833b-45e4-ac04-3cc2129cd6ea None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:44.429136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 306/622] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:44.430418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1284d2-63df-4762-a5c2-c05e7fdeff4b None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.430707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1284d2-63df-4762-a5c2-c05e7fdeff4b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.430981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1284d2-63df-4762-a5c2-c05e7fdeff4b None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.432356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 307/623] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:44.434424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1284d2-63df-4762-a5c2-c05e7fdeff4b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.449794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.450447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.450729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.451008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.451971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:44.452082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:44.453793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8757e7b8-e0ec-4b2e-a67a-f0da2f85979d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:44.470502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 317/624] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:44.470502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.470502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.470502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.484939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:44.486620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:44.557851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1284d2-63df-4762-a5c2-c05e7fdeff4b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.558353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 318/625] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:44.559190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.576254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:44.595297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.596456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.596981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.597483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.609868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:44.612231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:44.631937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:44.632965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:44.634003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:44.634003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:44.634003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:44.634003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:44.635812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 308/626] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:44.641256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.645364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:44.696602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263649f9-b80f-4abb-8ffc-00260b7a8f11 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.733563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.735030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.735314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.735588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/110d74df72174fba99493dedd2e98d08/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.751969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=110d74df72174fba99493dedd2e98d08, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:44.753673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:44.757351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f51113-ab63-4c95-be5e-32bfdd351371 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.777042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 319/627] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:44 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/110d74df72174fba99493dedd2e98d08/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:44.778314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.796336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.797428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.797714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.798250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.810854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:44.824373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:44.825397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:44.826275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 309/628] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:44.827169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:44.873509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:44.921044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78720a64-9acc-4724-8ba9-18d85c7aa143 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:44.967102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:44.968096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:44.968575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:44.969292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:44.970176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:44.970827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:44.972370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:44.987111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 320/629] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:44.991370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:44.993840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.080166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4e7c20-a352-45d9-86d3-c131b5a2c1d0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.126818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5507e144-dedf-4436-98f7-6063e30dbf04 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.129061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.129731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.130010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.130275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.131033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:45.131337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.132685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cea026-05ec-42dd-a0c0-fa587828b53f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:45.145665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.151057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 310/630] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:45.156982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3ca77-c53f-48d3-b261-96f0407a9530 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.157275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3ca77-c53f-48d3-b261-96f0407a9530 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.157584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3ca77-c53f-48d3-b261-96f0407a9530 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.158972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 311/631] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:45.159887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30beba41-c1cc-44d6-b279-3c4eaee95dcc None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.160166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30beba41-c1cc-44d6-b279-3c4eaee95dcc None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.160435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30beba41-c1cc-44d6-b279-3c4eaee95dcc None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.161971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 312/632] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:45 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:45.162707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30beba41-c1cc-44d6-b279-3c4eaee95dcc None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.172740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.173533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.173813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.174087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.181796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.183224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-e3183768-ba91-4d1e-90dc-1852e75d2362 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:45.197672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:45.198630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:45.199844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 321/633] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:51:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:45.200612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5ea225-edac-4495-9a06-18b831de1d3c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.200612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5ea225-edac-4495-9a06-18b831de1d3c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.200612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5ea225-edac-4495-9a06-18b831de1d3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.216612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-3a5ea225-edac-4495-9a06-18b831de1d3c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:45.217820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3a5ea225-edac-4495-9a06-18b831de1d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:45.278559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30beba41-c1cc-44d6-b279-3c4eaee95dcc None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.284222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 322/634] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:45.285524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.285810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.286083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.309061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:45.310242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:45.315693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.326502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.327560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.327838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.328292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.340678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.342116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:45.358790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:45.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:45.361614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:45.361614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:45.361614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:45.361614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:45.361614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 313/635] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:45.361614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.392966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 323/636] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:45.392966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.434671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.472059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d01719-4cf8-4bf2-9e13-ed43f6de714b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.504486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7fcb28-3261-4d4e-9600-24d0a62428fd None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.509798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.510667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.511151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.511598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.512607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:45.513141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.514726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:45.534030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 314/637] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:45.535071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.543659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.544450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.544754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.545034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.555926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:45.563802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:45.564741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:45.565579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 324/638] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:45.566250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.580155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.652115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca716b1-7a84-420f-9480-024c01947d0f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.670285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.677607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441f40da-3574-4ad8-b393-0f04a395d4e6 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.690419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.691394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.691682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.691958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.692680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:45.693016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.694369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:45.706835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 315/639] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:45.707640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.721712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.722502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.722783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.723056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.734917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:45.743045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:45.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:45.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:45.747159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 325/640] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:45.748045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.813974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.820142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5cf72-bbd3-4ed9-9b8a-fa144e4581ab None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.850962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d169c70-955a-4cd3-8770-6364d185132f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:45.853151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.853789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.854072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.854351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.889910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:45.891057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:45.891334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:45.891603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:45.892343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:45.892645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.894064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:45.907448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 326/641] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:45.907448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:45.924658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:45.934878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:45.936439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.011718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a6194c-9d71-49a7-9894-4737ed6d4cde None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.035144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.044609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 316/642] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:46.045512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.050615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.051407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.051686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.051957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.059949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.061475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:46.075706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:46.076826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:46.077779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 327/643] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:45 2022] GET /identity/v3/users/glance => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:46.079751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.153349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e166e1c9-8677-403a-853f-d4b7c77619c1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.161281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.186151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e01852-e8cc-47d3-8ee2-dbe36f0c0f25 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.188302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.189007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.189288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.189561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.190303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:46.190606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.191963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.200648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 317/644] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:46.201367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.226248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.227026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.227305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.227579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.239594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.241063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:46.258633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:46.259483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:46.260517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:46.260517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:46.260517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:46.260517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:46.260517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 328/645] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:46.260837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.266803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.311677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119c50c2-94aa-42de-bf4f-0d7ffa14dd6d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.346840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.347620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.347902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.348173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.358580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccca19c9-438d-40f0-b8ea-52c8eb174827 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.358964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.360552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:46.378762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:46.379642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:46.380515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:46.380515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:46.380515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:46.380515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:46.380515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 318/646] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:46.381383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.393208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.400054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.400743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.401037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.401298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.402129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:46.402434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.403836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.421401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 329/647] 10.222.0.33 () {56 vars in 1137 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:46.421761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.493274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a31559-ede7-4c51-9ca8-7ebb077e74be None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.505069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.526532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ea168-054e-4820-ad10-868bca5b950f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.530900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.531917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.532393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.532870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.544223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.545916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.560312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.561001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.561286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.561558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.562793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:46.563109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.564462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:46.566727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:46.567605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:46.568506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:46.568506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:46.568506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:46.568506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:46.568506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 319/648] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:46.568832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.578780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 330/649] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:46.579811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.635805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.683136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08da009-6704-4674-9d9d-f4fa756e1901 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.688447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096c391c-db9c-47c0-a2f7-0fee602e722f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.717976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.718662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.718982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.719214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.719932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:46.720240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.721751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.724094 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.724865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.725061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.725300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.737113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 320/650] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:46.737934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.769270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.801662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.803076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.852159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:46.854798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b44d8c-22ac-4c2b-a3c8-9c9f81aad17b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:46.893342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:46.894058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:46.894345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:46.894621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:46.895341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:46.895644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:46.897182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:46.913531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 331/651] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:46.913703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 321/652] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:46.913703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:46.914323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.008622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.021160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e7914a-7776-4f0a-ac87-c6ce8c06afe3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.027312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0006b8-9033-4539-a824-a573cf5ebef7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.055564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.056340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.056622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.056921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.061559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.062373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.062658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.063094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.063738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.065211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.069919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.071331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:47.079117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:47.080127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 332/653] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/users/cinder => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:47.081181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:47.085553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:47.086406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:47.087302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 322/654] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:47.087804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.097096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.190441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8280e892-0292-45c8-ab78-317702cb6a03 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.191927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33c46c9-f600-4328-9092-409e87864c40 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.223396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.224169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.224452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.224749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.225017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.225793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.226081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.226355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.231628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.233094 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.237090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.238517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.240225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d40ba6f2-7c3c-4335-8eb2-df6a23953dd6 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:47.249441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:47.250574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:47.251579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 333/655] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/users/neutron => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:47.252206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd5511-4184-4b9a-90aa-a2a9fd3cc285 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.252206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd5511-4184-4b9a-90aa-a2a9fd3cc285 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.252681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd5511-4184-4b9a-90aa-a2a9fd3cc285 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.253844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 334/656] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:47.254438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fd5511-4184-4b9a-90aa-a2a9fd3cc285 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:47.257466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:47.258503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:47.259623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:47.259623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:47.259623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:47.259623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:47.259623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 323/657] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:47.259623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.323565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.357284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fd5511-4184-4b9a-90aa-a2a9fd3cc285 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.362220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee1c60b-5c29-4f2a-b7f8-a097e124dce6 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.391824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.392475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.392787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.393065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.393906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:47.394215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.395280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.395633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec11296-98db-4586-8d48-37ef8b8774b9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.395926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.396209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.396486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.397303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:47.397614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.399019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.410788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 324/658] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:47.411623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.422265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 335/659] 10.222.0.33 () {56 vars in 1137 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:47.422265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.422265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.422265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.435882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:47.437107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:47.474088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.507497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 336/660] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:47.508470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.518028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f9666a-7f91-40cd-8e3b-69f92cea41a9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.550972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.551635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.551916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.552186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.553041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:47.553346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.554754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.572033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 325/661] 10.222.0.33 () {56 vars in 1139 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:47.572871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.574389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.619696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61842f4-86ee-4817-8677-03ce122bf0cc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.653061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.653709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.654257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.654969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:47.655272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.657225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.668393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 337/662] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:47.669205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.688277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6de5c-8db6-4e59-83a8-f6ae58cb2790 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.701546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.724210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.724831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.725121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.725394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.726225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2a1c13591bab47d7ba589e14b83f969e) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:47.726538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.727953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.733647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 326/663] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/role_assignments?user.id=2a1c13591bab47d7ba589e14b83f969e&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:47.734416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.779945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a34e10-fa77-4def-b7e1-a827e6f50796 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.800014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:47.823319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.823991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.824276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.824549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.850880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38f1744-5816-4d8c-a574-a5fcef032f3a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:47.894936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:47.896294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:47.896294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:47.896480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:47.900163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:47.901724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:47.908596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:47.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:47.917907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:47.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:47.921259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 327/664] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:47.922322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:47.974156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.016435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 338/665] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:48.017638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.038743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.056400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e161a-1b2e-48b7-9647-80e45ee484b7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.091016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.091652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.091933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.092211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.131270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60035e9f-21fd-4da1-ad7e-790d2509afcd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.165102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.166608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.167447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.168237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.168629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.168928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.176798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.178221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:48.192534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:48.193583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 339/666] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:48.194465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.210189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.276259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.290050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 328/667] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:48.290995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.302780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9a802a-d4c9-402e-a7c9-b97bb6013f9a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.338764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.339442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.339715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.339992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.340902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:48.341330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.342702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.355948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 340/668] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:48.355948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.399122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9346583-4efc-409a-bafe-a3c46386fd02 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.434280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.435068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.435355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.435808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.439417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.448441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.449936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.460489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addddc54-ba04-4894-a237-92c2d9fb296e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:48.468399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:48.469816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:48.470698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:48.470698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:48.470698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:48.470698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:48.470698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 329/669] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:48.470970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.500890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.501666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.501953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.502239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.503010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:48.503319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.505064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.513385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.523895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 341/670] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:48.525155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.579625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb5955b-327c-4896-a5f9-08aae75ceac5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.616257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.617144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.617481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.617754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.630656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.640319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f4b8b-f475-4a55-9fcf-61fd8c2432c2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:48.649577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:48.650461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:48.651404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:48.651404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:48.651404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:48.651404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:48.651404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 330/671] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:48.652123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.691641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.692457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.693299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.693583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.694902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.704835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.706292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:48.723421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:48.724367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:48.725288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:48.725288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:48.725288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:48.725288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:48.725288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 342/672] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:48.726476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.763898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084dce09-1963-467c-a64c-d7f1ef605181 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.783637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.799682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.800350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.800646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.800946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.801739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:48.802048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.803466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.819594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 331/673] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:48.819594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.843585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044372cf-35a9-402d-be37-37eaa7204097 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:48.882814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:48.883466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:48.883749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:48.884020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:48.884894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=9f86b27721074978b12f34dd591362ce) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:48.885206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:48.886619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:48.892733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 343/674] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/role_assignments?user.id=9f86b27721074978b12f34dd591362ce&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:48.893253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:48.984097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:48.994809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8af9bb5-fb0e-40a3-9e9c-c1f4ce1580b9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.050102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.051153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.051652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.052136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.053103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.053618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.058333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.073098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 332/675] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:49.074398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.102913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2d7973-c715-47ca-8fba-658d98134c87 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.121462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:49.149045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.149491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.149776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.150050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.150775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.151079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.153943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.168992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 344/676] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:49.170091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.205470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1125fd6-abae-48fe-b266-9aa6d56b101c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.248725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.249551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.249871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.250147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.258087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.259553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:49.273793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:49.274721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:49.275603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 333/677] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:49.276081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.320115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:49.339874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2059f25d-f67f-4c53-9d2c-57a4a22435cd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.414846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.415928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.415928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.416121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.424844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.426538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.434576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4121779f-e904-4517-b7fb-1c01e03ffc3a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:49.441692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:49.442595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:49.443495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:49.447821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 345/678] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:49.448555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.481665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.482902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.483372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.483837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.484439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:49.494231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.496647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:49.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:49.517938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:49.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:49.519434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 334/679] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/users/swift => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:49.520122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.568628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff6dd9a-da7f-45d1-8499-c6717bceae39 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.602907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.603592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.603874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.604146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.604965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.605270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.606707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.625679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 346/680] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:49.626260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:49.626543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.652308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6dd78a-847a-4bdc-92ad-0357dbe9f160 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.694633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.695317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.695670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.697005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.697242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.698662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.716582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 335/681] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:49.717538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.745201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:49.753233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a4481e-ea49-4493-a792-3694d22b07f8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.797824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.799411 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.799904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.800395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.801607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.803136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.804576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.829418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 347/682] 10.222.0.33 () {56 vars in 1135 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:49.830520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71054601-0a09-4c99-90fd-e31225a3e44a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.830824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.880484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.881170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.881455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.881894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.883100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=102f9668a5f948f99ac4ba6fc639c8ad) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.883547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.885280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:49.901170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 336/683] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/role_assignments?user.id=102f9668a5f948f99ac4ba6fc639c8ad&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:49.902295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:49.906225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:49.943849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04edeeac-bf0a-4f51-a4f1-f4adac4181bd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:49.983682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:49.984842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:49.985156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:49.985439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:49.986617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=ea109e86f77f4660b6538e0ec4676629) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:49.986971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:49.988456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6fe8da-2ddb-477a-91aa-7449d7d48628 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:50.000725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 348/684] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/role_assignments?user.id=ea109e86f77f4660b6538e0ec4676629&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:50.000725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb8a32-0ed3-4b8c-b268-9ecaf6e6b019 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.000725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb8a32-0ed3-4b8c-b268-9ecaf6e6b019 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.000725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb8a32-0ed3-4b8c-b268-9ecaf6e6b019 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.000725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 349/685] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:50.005128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.005436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.006114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.031775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:50.033083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:50.059491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:50.082287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaa8ea3-d232-49f2-a7ab-9ac105b60095 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:50.137702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:50.138675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.138963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.139767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.145126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 350/686] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:50.170615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:50.227010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:50.228558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:50.231384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:50.309342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f842159d-264a-4a1d-b0ad-050d6b74c84e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:50.348796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:50.349593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.349769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 337/687] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:50.349892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.350129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.358017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:50.378700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:50.387724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:50.388729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:50.389561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:50.390206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 351/688] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:50.396194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:50.413547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:50.566208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcda6398-b1dd-435f-b7aa-5ee9f687ae66 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:50.571673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:50.594683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb38faf5-8209-4bc0-b1d6-c6b6a6e7d8bd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:50.620439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:50.621625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.623274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.624616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.647719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:50.649216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:50.658265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:50.658936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.659219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.659494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.660248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:50.660553 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:50.661942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:50.675098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:50.676030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:50.676956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:50.676956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:50.676956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:50.676956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:50.678065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 338/689] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:50.687217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:50.691977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 352/690] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:50.698123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:50.822482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:50.853620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5374579-c58f-421f-9ca9-c8082002ded8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:50.871171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e4082-bac3-40b3-a35c-f9547a249665 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:50.918739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:50.920577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.922452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.922941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.925104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:50.925644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:50.928058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:50.943398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:50.944197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:50.944480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:50.944783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:50.961208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 339/691] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:50.966252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:50.967711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:50.969593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:50.997631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:51.008844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:51.009817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:51.010702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:51.010702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:51.010702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:51.010702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:51.016727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 353/692] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:51.018374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.127916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57b1ce-be95-4b32-a7e1-cac5f0c7d336 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.165545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.166332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.166616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.166886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.175257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.177131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:51.193965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:51.194886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:51.195772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 340/693] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:51.200800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.232404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe40fa6a-a949-4faa-b260-45127e89e08b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.267198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.283364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.284042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.284325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.284602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.285511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:51.285826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.287379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8314b2c9-1568-4d5c-9422-ae0f87844cae None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.294841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.304475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 354/694] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:51.305865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254f0f37-b3f8-4650-847a-daa089a7c48c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.306155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254f0f37-b3f8-4650-847a-daa089a7c48c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.306428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254f0f37-b3f8-4650-847a-daa089a7c48c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.311959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 355/695] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:51.313368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254f0f37-b3f8-4650-847a-daa089a7c48c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.330344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba66288-38c9-49b2-bf63-4fb21f4ea533 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.371738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.372504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.372815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.373092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.373885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:51.374191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.376400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c653226-4b0c-43b9-8762-31db86f2d61b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.389359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 341/696] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:51.390655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.390949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.391223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.408069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:51.409295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:51.435308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254f0f37-b3f8-4650-847a-daa089a7c48c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.480166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.480995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.481278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.481552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.488960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 342/697] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:51.489818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.489989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.491256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:51.507729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:51.509920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:51.512122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 356/698] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:51.520718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.563556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.589177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.632198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7963d-982f-4d5e-971d-7f438a00c240 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.633178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37092463-6240-49b7-8285-7ec08ae718c8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.674827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.675477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.675669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.675820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.676034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.676394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.676763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.676906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:51.677196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.677347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.678179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=110d74df72174fba99493dedd2e98d08) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:51.678554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.678701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.680075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.688491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 343/699] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/role_assignments?user.id=110d74df72174fba99493dedd2e98d08&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:51.689354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.697942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 357/700] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:51.704174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:51.834092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3060e1f-59d6-492c-bc20-772397c1110f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.842642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ddc4b-e3ec-4a67-8fe7-5f0c87c42a28 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:51.845212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.905715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:51.914176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.915106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.915440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.915785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.923779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:51.924444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:51.924751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:51.925084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:51.930403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:51.932017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:51.951818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:51.953467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:51.954362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:51.954362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:51.954362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:51.954362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:51.956725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 344/701] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:51.960651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.020160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.021896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.076219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d251c79-b6fe-4842-afbe-46fb19fa7876 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.115822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.116792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.117227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.117643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.118557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:52.119054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.121459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.129009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:52.134800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 358/702] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:52.138190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 345/703] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:52.140963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.146156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.202453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:52.260144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46ac7d1-91b7-4d15-bd24-ec314ffcb56a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.289281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fda684-e46e-4fc6-b129-e7e70086326c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.299337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.300315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.300719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.301106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.314413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.316013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.334665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331ade24-bc0b-4f2d-9b4f-8ba63c8f3cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:52.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:52.336244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:52.337347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:52.337347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:52.337347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:52.337347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:52.338342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ade24-bc0b-4f2d-9b4f-8ba63c8f3cc9 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.338771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ade24-bc0b-4f2d-9b4f-8ba63c8f3cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.339178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ade24-bc0b-4f2d-9b4f-8ba63c8f3cc9 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.340011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331ade24-bc0b-4f2d-9b4f-8ba63c8f3cc9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.341598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331ade24-bc0b-4f2d-9b4f-8ba63c8f3cc9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.343170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 359/704] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:52.344502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.371139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 346/705] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:51:52 2022] POST /identity/v3/users => generated 254 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:51:52.410940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:52.422393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e5a398-e9ef-4657-919d-9ca303446c14 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.423022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e5a398-e9ef-4657-919d-9ca303446c14 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.423526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e5a398-e9ef-4657-919d-9ca303446c14 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.425463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 347/706] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:52.434219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0fdaf-be8c-4e5b-9785-c099909fc884 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.434861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0fdaf-be8c-4e5b-9785-c099909fc884 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.435339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0fdaf-be8c-4e5b-9785-c099909fc884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.467540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f6a0fdaf-be8c-4e5b-9785-c099909fc884 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:52.469170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f6a0fdaf-be8c-4e5b-9785-c099909fc884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:52.470728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79476e5c-68ef-4f46-b03f-237f4665c509 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.509402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.510077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.510364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.510641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.511381 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:52.511688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.513115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1f22e0-0340-4951-8b09-c3a9daf51786 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.530898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 360/707] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:52.532602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe300a9-acf7-4149-b051-b2697202841f None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.532937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe300a9-acf7-4149-b051-b2697202841f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.533218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe300a9-acf7-4149-b051-b2697202841f None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.535156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 361/708] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:52.536066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe300a9-acf7-4149-b051-b2697202841f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.545357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 348/709] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:52.546757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.547051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.547327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.565006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:52.567089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:52.592203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:52.653437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 349/710] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:52.654364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.663162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe300a9-acf7-4149-b051-b2697202841f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.677532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:52.719168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.720798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.721226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.721641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.732476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.733971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:52.757127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:52.758096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:52.758970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:52.761962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 362/711] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:52.762879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.785605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65790932-fec9-465a-b5af-99835a3f0ebb None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.828754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.830151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.830578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.831003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.832489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.834039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:52.854275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:52.855420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:51:52.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:52.860944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 350/712] 10.222.0.33 () {56 vars in 1129 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:52.860944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.891890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:52.892858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6915068f-4521-4ea8-80d4-7249bd4c76fd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:52.937127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:52.939071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:52.939354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:52.939627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] PATH_INFO: `/v3/services/network` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:52.940285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:52.941990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:52.961117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:52.962154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:52.963398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 363/713] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/services/network => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:52.964181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:52.994819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157b994e-12a4-42b9-99be-a93306f991f9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:53.010984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:53.038829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:53.039961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:53.040248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:53.040531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:53.041635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:53.041948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:53.044014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:53.068167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 351/714] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:53.068167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:53.089767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d9fe9-1511-4f7b-a462-437895997b0a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:53.130817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:53.132027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:53.132487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:53.133084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:53.134275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:53.135012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:53.137033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:53.142991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 364/715] 10.222.0.33 () {56 vars in 1124 bytes} [Tue Jul 19 10:51:52 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:53.143988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:53.169727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:53.194963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051d4251-f55b-4039-ac7a-efc99049638e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:53.243197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:53.244163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:53.244625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:53.245152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:53.246051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] RBAC: Adding query filter params (name=network) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:53.246550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:53.248082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:53.253581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 352/716] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:51:53 2022] GET /identity/v3/services?name=network => generated 124 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:53.254667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:53.270511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:53.275403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b7beb2-0516-4b74-98e8-5ad3f122bcb0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:53.312825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:53.313892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:53.314402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:53.314878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:53.315932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2a1c13591bab47d7ba589e14b83f969e) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:53.316446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:53.318169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:53.324389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 365/717] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:53 2022] GET /identity/v3/role_assignments?user.id=2a1c13591bab47d7ba589e14b83f969e&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:53.325459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:53.408244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cda82-b54b-4759-8cdf-9d373c333d6d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:53.497085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:53.497721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:53.498026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:53.498315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:53.498987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:53.499291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:53.500766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9645f0c9-a70a-4453-9a4f-a0fd201bc949 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:53.509038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb1955d-a9c3-4afe-bb3b-fa71452624d1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:53.510099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 353/718] 10.222.0.33 () {56 vars in 1124 bytes} [Tue Jul 19 10:51:53 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:53.552369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:53.553484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:53.553975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:53.554448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:53.555397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] RBAC: Adding query filter params (type=network) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:53.555926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:53.557587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d278def3-f7d3-4e56-9781-e9efa6076804 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:53.564605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 366/719] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:51:53 2022] GET /identity/v3/services?type=network => generated 124 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:54.330987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe709f94-766f-4e0b-bba9-20b850a32286 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:54.331184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe709f94-766f-4e0b-bba9-20b850a32286 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:54.331462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe709f94-766f-4e0b-bba9-20b850a32286 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:54.333122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 354/720] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:54.338477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:54.338933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:54.339010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:54.360744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:54.362211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:54.393286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:54.461271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 367/721] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 19 10:51:54.510156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe709f94-766f-4e0b-bba9-20b850a32286 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:54.524117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:54.644597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe709f94-766f-4e0b-bba9-20b850a32286 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:54.679391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b758f9b-9a93-405b-9eeb-174793c2a1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:54.680171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b758f9b-9a93-405b-9eeb-174793c2a1e9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:54.680453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b758f9b-9a93-405b-9eeb-174793c2a1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:54.680743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b758f9b-9a93-405b-9eeb-174793c2a1e9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:54.691645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b758f9b-9a93-405b-9eeb-174793c2a1e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2b758f9b-9a93-405b-9eeb-174793c2a1e9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:54.702578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:54.704546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:54.707307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 355/722] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:54.712421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:54.790546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943c687-2562-463f-aa02-fdcb99694c68 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:54.790877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943c687-2562-463f-aa02-fdcb99694c68 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:54.791108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943c687-2562-463f-aa02-fdcb99694c68 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:54.792951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 356/723] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:54.798570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:54.798892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:54.799124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:54.830030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:54.831224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:54.860473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3e3dac-73c0-4b33-bd7a-a3d42e7bb496 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:54.897969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:54.906847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 357/724] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:54.924659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:54.925380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:54.925665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:54.927267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:54.928030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:54.928340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:54.929747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:54.947321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 368/725] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:54.952771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:54.961773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.006240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:55.062958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0574f7-ffb9-4390-b356-0be99a37c8bd None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.098500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.099293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.099575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.110933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:55.112632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:55.131573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:55.133731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:55.135857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:55.135857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:55.135857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:55.135857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:55.139925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 358/726] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:55.139925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.199677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:55.249328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37be3c8-c294-4b7a-a41e-092113c16f6a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.278655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8984f1d-04af-48fe-86be-fabd381c935a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.291133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.291813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.292098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.292378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.295558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:55.295879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:55.297837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:55.322578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 359/727] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:55.332088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.332927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.333214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.333486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.333692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.347995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:55.360056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:55.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:51:55.364431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:55.370072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 369/728] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:51:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 406 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:55.371761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.514485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724e52b-245d-45e4-ae29-391d729e955f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.531698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:55.544610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:55.573649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6287f72-f2e0-4933-88bd-70002dd775a3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.592904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.593752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.594041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.594317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.610390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:55.612112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:55.622627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.623300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.623586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.623944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.624727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:55.625643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:55.627092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:55.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:55.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 360/729] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:55 2022] GET /identity/v3/users/swift => generated 83 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:55.642423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.648732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 370/730] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:51:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:55.650917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.804886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58d7895-dfb3-4384-8fa1-06b44a8e419e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.851438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5406fd-95f9-493e-8859-37da3843c78b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:55.854723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.855643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.856128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.856598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.857663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:55.858197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:55.860354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ec50bc-7171-4862-b83b-28d5a4dcf3b8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:55.881684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 361/731] 10.222.0.33 () {56 vars in 1135 bytes} [Tue Jul 19 10:51:55 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:55.883287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8c757f-0237-4785-b766-2b69257316d1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.883675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8c757f-0237-4785-b766-2b69257316d1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.883954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8c757f-0237-4785-b766-2b69257316d1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.885448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 362/732] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:55.887089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8c757f-0237-4785-b766-2b69257316d1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:55.888436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:55.889264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.889551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.889919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.899536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:55.902225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:55.903963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-1575fdba-8593-44e9-b6bd-01dc0e03d2ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:55.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:55.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:55.925339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:55.925339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:55.925339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:55.925339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:55.925339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 371/733] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:55.926354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:55.926645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:55.926923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:55.932157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:55.952632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:55.954308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:56.020076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8c757f-0237-4785-b766-2b69257316d1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.026969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 372/734] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:56.027926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.062054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.063153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.063435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.063708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.130554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c12e96f-e32a-4233-94cb-651950733d5b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.135363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.136858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.154645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.167581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.168245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.168530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.168831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.169561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:56.169875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.170067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.171293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432ede0-7c60-469e-bfc0-c45657ef403f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.183382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 373/735] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:56.184677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db58f4-a595-4367-aa87-291f7d4717cb None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.185008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db58f4-a595-4367-aa87-291f7d4717cb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.185284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db58f4-a595-4367-aa87-291f7d4717cb None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.190892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 374/736] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:56.191653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db58f4-a595-4367-aa87-291f7d4717cb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.247278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 363/737] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 19 10:51:56.248428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.297963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db58f4-a595-4367-aa87-291f7d4717cb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.331240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e840ff-3fd4-49a3-8daf-094e6864e487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.331946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e840ff-3fd4-49a3-8daf-094e6864e487 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.332225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e840ff-3fd4-49a3-8daf-094e6864e487 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.332500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e840ff-3fd4-49a3-8daf-094e6864e487 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.342325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e840ff-3fd4-49a3-8daf-094e6864e487 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:51:56.346903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c267f426-67d1-45fe-b852-94efef1a4071 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-b7e840ff-3fd4-49a3-8daf-094e6864e487 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:51:56.349903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:56.351308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:51:56.352126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:56.355884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 375/738] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:56.357161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.357450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.357730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.374409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:56.376145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:56.376589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.382698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.383500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.383785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.384178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.391185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.392617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2df7da3e-e077-4dfb-a2dd-156e5a887991 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:56.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:56.408381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:51:56.409372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:56.411707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.412901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 364/739] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:56.414512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06349dc-9d02-4a54-82b3-8d5f95436eac None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.414871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06349dc-9d02-4a54-82b3-8d5f95436eac None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.415163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06349dc-9d02-4a54-82b3-8d5f95436eac None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.416455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 365/740] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:56.417171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06349dc-9d02-4a54-82b3-8d5f95436eac None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.447588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 376/741] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:56.448408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.518176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06349dc-9d02-4a54-82b3-8d5f95436eac None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.543762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8de0a11-1833-4cf6-ade8-cfc742f2e5c3 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.551058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.551836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.552119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.552392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.563036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.564469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.577253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.577904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.578214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.578484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.579217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:56.579515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.580901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec9ac4-69dc-4382-bc17-8f287dd60430 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:56.581531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:56.582522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:56.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:56.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:56.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:56.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:56.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 366/742] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:56.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.591001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 377/743] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:56.592264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.592555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.592860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.608464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:56.610192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:56.632007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.637942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.677898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 378/744] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 19 10:51:56.678695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.703914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa57e3-5ed9-417c-8fea-4f1bce2cc7ad None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.742582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.743259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.743543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.743820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.744663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:56.745000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.746439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.763520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 367/745] 10.222.0.33 () {56 vars in 1149 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:56.764323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.776840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25949ac-7fd0-4ddf-aa09-45ecb1867a93 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.810513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.811170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.811454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.811730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.812443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.813841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.836307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 379/746] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:51:56 2022] POST /identity/v3/services => generated 240 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:56.836307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.864704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.876102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:56.879396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2bccfa-09ca-415e-b749-8594595dc787 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:56.926976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:56.927713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:56.928003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:56.928278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:56.929056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:56.932247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:56.932247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:56.949204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 368/747] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:56.950280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:56.962187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de908b38-c101-4b57-8994-0bf82fc35f7b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.002382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.003205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.003494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.003770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.019536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.021024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:57.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:57.040578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:57.041608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:57.041608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:57.041608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:57.041608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:57.041608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 380/748] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:57.041952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.066328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c039b-b939-4edd-9958-3778f32faee7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.104767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.105422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.105693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.105966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.106739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.108133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.132730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 369/749] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:51:56 2022] POST /identity/v3/services => generated 251 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:51:57.132730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.136315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.166837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4efcba3-f689-44ca-8b1e-37a12b2b6d78 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.173087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.225540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.226387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.226842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.227221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.277568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c18b107-fbd2-49b5-bc66-bb2fb51a2a2b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.323939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.325479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.331158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.331671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.332154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.340884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.343147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:57.358610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:57.359741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:57.360868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:57.363377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 370/750] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:57.364360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.403655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.441311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 381/751] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:51:57.442449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.472182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.504222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b286e907-e99b-4752-88a2-2da2e267bad8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.551270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa5bf69-ff98-474d-b0e2-4297d31b6bb2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.551630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.552429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.552848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.553229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.554319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:57.554728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.557113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.572803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 371/752] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:57.573874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.599641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.600577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.600998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.601379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.602391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:57.602799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.604454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.622544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 382/753] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:57.622544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.655514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.701938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5462f401-3566-4485-8d4d-befbe7a1ed02 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.734039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f26fc-e2aa-4a81-8c12-a22adad62e59 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.745479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.746804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.747369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.747819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.752000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.764789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.767652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.776935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.778160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.778671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.779147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.787164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:57.787361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:57.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:57.789143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:57.789143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:57.789143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:57.789143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:57.791447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.793203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 372/754] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:57.794035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:51:57.808259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:57.809332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:51:57.810296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:57.818089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 383/755] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:57.819189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:57.922657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:57.923834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d579f4-8b4f-496c-ba8f-13915d7e76ce None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.934321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351be08b-27a9-4fdd-bb30-e11fa50c0e38 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:57.971535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.972053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:57.972460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.972883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.973242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:57.973357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.973763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:57.974180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:57.974475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=110d74df72174fba99493dedd2e98d08) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:57.974890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.975765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:57.976217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:57.976617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.978120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:57.988770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 373/756] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/role_assignments?user.id=110d74df72174fba99493dedd2e98d08&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:57.988770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.007067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 384/757] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:58.012474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.056554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:58.115435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02b3d4-2220-482a-8bd7-e655aba86743 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.156224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.156952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.157245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.157527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:58.159048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:58.159368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:58.161032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:58.187149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 374/758] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:51:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:58.188637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.278376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d194083-b553-4ee0-bc83-c6d35cec72db None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.340679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6834e8bb-1692-4ce6-b3f7-da11641b0530 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.349998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.351083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.351584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.352066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:58.362868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:58.364591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:58.365033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:58.380147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:58.381380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:58.382278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:51:58.393066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 385/759] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:58.397362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.422510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.423687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.424194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.424660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:58.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:58.447214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:58.449085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:51:58.480949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:58.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:58.482778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:51:58.482778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:51:58.482778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:51:58.482778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:58.488894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 375/760] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:51:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:51:58.494655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.571995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dfdd90-f3a9-489e-bf50-2441852acf2d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.630521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.631575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.632036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.632483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:58.640715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:58.640715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:58.640715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:58.661067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 386/761] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:58.668344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.673795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676c86ac-d1b9-4948-9915-9babb03b77f3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.750616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.751214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.751497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.751767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:58.752494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:58.752822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:58.754189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:58.767385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 376/762] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:51:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:58.772750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:58.773710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:58.804406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ff428-d7d3-4204-8718-b7e41e2259d3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.805258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:58.853000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.854248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.855565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.856023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:58.857071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2781978834a04fd4a48b4fb2ae2892d7) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:58.857579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:58.859188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:58.868159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 387/763] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:51:58 2022] GET /identity/v3/role_assignments?user.id=2781978834a04fd4a48b4fb2ae2892d7&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:58.920476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b590df-524c-48c8-b66f-a4d201bfdbb1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:58.990202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:58.991006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:58.991291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:58.991566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.004387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:59.005829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-7806d57d-d97a-465b-9a51-30a607f85d47 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:51:59.026229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:59.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:59.028096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 377/764] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:59.034779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.053724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6827c136-e270-429c-8a86-866ff15b4f20 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.054057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6827c136-e270-429c-8a86-866ff15b4f20 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.054296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6827c136-e270-429c-8a86-866ff15b4f20 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.056063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 378/765] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:51:59.061335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.061628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.061935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.080686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:59.082060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:59.185878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 379/766] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:59 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:51:59.205891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcd599-82bc-44b6-a0c8-f1f7bbc2d862 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:59.227184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.258340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:59.282525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:59.283194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.283475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.283748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.284614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:59.284858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:59.286284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:59.300159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 388/767] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:51:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:59.301050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:59.306043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.436219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435611ce-c813-469c-81f4-9a80443dec00 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:59.473137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:59.474237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.474533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.474816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] PATH_INFO: `/v3/services/network` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.475679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:59.477262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:51:59.491235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:51:59.492274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:51:59.493554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 380/768] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:51:59 2022] GET /identity/v3/services/network => generated 88 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:51:59.499378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.546568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46880d03-27b1-4ffb-a91d-d91fd58e87bf None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:59.581784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914cebe7-6332-4148-82d0-c77936861b4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:59.582673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914cebe7-6332-4148-82d0-c77936861b4a None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.582957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914cebe7-6332-4148-82d0-c77936861b4a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.583229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914cebe7-6332-4148-82d0-c77936861b4a None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2a1c13591bab47d7ba589e14b83f969e/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.584027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:59.585254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914cebe7-6332-4148-82d0-c77936861b4a None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=2a1c13591bab47d7ba589e14b83f969e, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:59.586679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914cebe7-6332-4148-82d0-c77936861b4a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:59.602527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 389/769] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:51:59 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2a1c13591bab47d7ba589e14b83f969e/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:51:59.603970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f306bb2e-460a-47e2-8894-e43e69d70bf2 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.604309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f306bb2e-460a-47e2-8894-e43e69d70bf2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.604644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f306bb2e-460a-47e2-8894-e43e69d70bf2 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.611017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 390/770] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:51:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:51:59.613408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.613818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.614190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.622457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd6dbe-101e-49a3-9b34-7fa6dbd10ad1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:59.634201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:51:59.635778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:51:59.662531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:59.663500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.663940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.664450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.665595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] RBAC: Adding query filter params (name=network) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:59.666102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:59.667691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:59.674377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 381/771] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:51:59 2022] GET /identity/v3/services?name=network => generated 124 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:51:59.681070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.699938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:59.721028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 391/772] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:51:59 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:51:59.771919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.821505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1d000c-804b-4ddc-9f8a-a5a1f5678cdb None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:59.861343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:51:59.862798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:51:59.863292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:51:59.864392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:51:59.865386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] RBAC: Adding query filter params (type=network) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:51:59.865923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:51:59.867510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:51:59.873592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 382/773] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:51:59 2022] GET /identity/v3/services?type=network => generated 350 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:51:59.879402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:51:59.884087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:51:59.956058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e19e61-34f1-4a69-a765-327bc0006582 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:51:59.997391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f59ff-5c25-47f5-8aa7-c5effc10f1a5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.003576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.004668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.005188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.006191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.007081 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.009205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:00.026711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:00.027672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:52:00.028492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:00.030220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 392/774] 10.222.0.33 () {56 vars in 1129 bytes} [Tue Jul 19 10:51:59 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:00.036163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:00.046036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.047070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.047559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.048056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.049081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] RBAC: Adding query filter params (interface=public, service_id=4b61dcd459e54ed6b542341e9cd8b2d6, region_id=RegionOne) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:00.049598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.051220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.060816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 383/775] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:51:59 2022] GET /identity/v3/endpoints?service_id=4b61dcd459e54ed6b542341e9cd8b2d6&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:00.066250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:00.117795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:00.173774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb406ce-21a5-487c-97b4-c5ab41d737ae None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.195082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:00.212866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.213903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.214400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.214874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.215889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:00.216396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.218058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.224358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 393/776] 10.222.0.33 () {56 vars in 1124 bytes} [Tue Jul 19 10:52:00 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:00.230089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:00.251709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd57225-da5e-4c52-81f1-2a78c2692cfd None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.301726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.302423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.302704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.302979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.303664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.305052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.317063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 384/777] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:00 2022] GET /identity/v3/services => generated 772 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:00.347829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091e3dae-b9ff-4910-84d3-22b8ea584ed9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.364554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:00.388408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.389444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.389931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.390432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.391957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:00.392552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.394213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239dad7-e89e-4cac-b3f9-256d6fd0e7b3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.404813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 394/778] 10.222.0.33 () {56 vars in 1124 bytes} [Tue Jul 19 10:52:00 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:00.407688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:00.520366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70e46d-2669-449b-948a-3d1294b22361 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.521021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70e46d-2669-449b-948a-3d1294b22361 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.521505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70e46d-2669-449b-948a-3d1294b22361 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.523426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 395/779] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:00.528798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.529332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.529824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.538742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e0db99-1e7b-4f96-8f66-e456c13dda27 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.549027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:00.550230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:00.595686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:00.608396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.609100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.609420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.609697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.610505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] RBAC: Adding query filter params (interface=public, service_id=9328cfac63c94379bc93c37e51a06cdf, region_id=RegionOne) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:00.610862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.612751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.618820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 385/780] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:52:00 2022] GET /identity/v3/endpoints?service_id=9328cfac63c94379bc93c37e51a06cdf&interface=public®ion_id=RegionOne => generated 194 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:00.624366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:00.681279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 396/781] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:00.752506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:00.759893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb84c49-c361-4a94-bfcd-0f424e277296 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.804817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:00.805542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:00.805990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:00.806403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:00.807097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:00.808734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:00.815865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 386/782] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:00 2022] GET /identity/v3/services => generated 772 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:00.956288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0e1e25-ceaf-4e1c-8cf3-87f43d1a460b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:00.975201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:01.013320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:01.014119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.014406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.014681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.027308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:01.035196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:01.036117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:01.036967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:01.039637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 397/783] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:01.045635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.111010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:01.155273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d994cb-4645-49aa-8cd9-5a1429c99da5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:01.196020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:01.197148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.197687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.198163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.199122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:01.199625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:01.201248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6505e91d-8151-4817-a980-7cca4284caed None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:01.212985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 387/784] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:01.219343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.245669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ed6b8-e974-4657-acac-dd53a1a7e6f7 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.246284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ed6b8-e974-4657-acac-dd53a1a7e6f7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.246760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ed6b8-e974-4657-acac-dd53a1a7e6f7 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.248605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 388/785] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:01.253743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.254292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.254768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.287325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:01.290394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:01.366910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d911b186-f121-4b64-9021-c5134965ba26 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:01.404547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:01.405770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.406316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.406792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.415405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 389/786] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:01 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (3 switches on core 0) Jul 19 10:52:01.418787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:01.420448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:01.438758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:01.439655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:01.440564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:01.440564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:01.440564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:01.440564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:01.442714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 398/787] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:01.448104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.454682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.493818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:01.537863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:01.620503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf49f45-8859-483d-86c1-bedf48e8a684 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:01.647361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918c8c63-eaa2-4633-9e45-ff3daeb14267 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:01.697868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:01.698574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.698804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.699080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.699823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:01.700112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:01.701514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:01.703544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:01.704649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.705156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.705629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.716711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:01.725028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:01.725956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:52:01.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:01.727423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 390/788] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:01.729546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 399/789] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:01.729730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.735330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.834782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:01.857615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2762a77-b7f2-4a0a-b5a3-c5ca08c1ef60 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:01.895143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:01.896170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:01.896684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:01.897170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:01.897963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:01.898286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:01.899669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:01.912722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 400/790] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:01.914898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:01.987523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ed8a1-e291-4601-acca-2b8be5936d3e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.017054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.061510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.062310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.062628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.062905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.081119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.082581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:02.115214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:02.116228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:52:02.117217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:02.127227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 391/791] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/users/swift => generated 83 bytes in 398 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:02.130294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922e223f-09b4-463d-9d48-63c88c199361 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.138596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.214008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.214811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.215098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.215382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.241662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.243044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.246657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.255886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda26b77-a013-46da-940f-2e721a9450b4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:02.274437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:02.275593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:02.276490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:02.276490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:02.276490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:02.276490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:02.285150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 401/792] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:02.290698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.308280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.309373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.309870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.310353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.311413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:02.311944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.314589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad205b3f-7f97-4255-ac66-626d02af40f0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.342105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 392/793] 10.222.0.33 () {56 vars in 1135 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:02.349023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d462d9-cc0c-4ebd-88ac-bb7e9ba93572 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.349871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d462d9-cc0c-4ebd-88ac-bb7e9ba93572 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.350362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d462d9-cc0c-4ebd-88ac-bb7e9ba93572 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.352744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 393/794] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:02 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:02.353844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d462d9-cc0c-4ebd-88ac-bb7e9ba93572 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.391813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.479349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d462d9-cc0c-4ebd-88ac-bb7e9ba93572 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.525287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.526526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.527373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.527980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.541832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.543658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.547533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ede21-1cba-49cc-8a07-fc6119c8b65f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-9a5cc550-4053-41d7-92c8-8caf552fea8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:02.567520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:02.568612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:02.569746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:02.569746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:02.569746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:02.569746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:02.576636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 394/795] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:02.576636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.576636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.576636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.594777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.595767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.596363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.596935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.597043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:02.597962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:02.598345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:02.598687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.600278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.616561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 402/796] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:02.617967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.684126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 395/797] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:02 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:02.685171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.693384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.714971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.741123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cad026-5189-4b01-a36e-aa951a31b141 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.782337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.782922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.783203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.783474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.784196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:02.784502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.785906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.798487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 403/798] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:02.799004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.818029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f1c83-f690-46a4-93ff-3c2cb2d9dc6b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.856089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.856940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.857237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.857510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.865855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:02.867301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-5a6e1f70-3dae-4d62-be4e-012309d8575e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:02.882818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:02.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:02.884733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 396/799] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:02.885818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e330e-4feb-4fd2-9b8b-cc0e4e827f01 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.885925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e330e-4feb-4fd2-9b8b-cc0e4e827f01 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.886365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e330e-4feb-4fd2-9b8b-cc0e4e827f01 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.887561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 397/800] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:02.891763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943e330e-4feb-4fd2-9b8b-cc0e4e827f01 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:02.916084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a434f7a-ef55-4616-84da-0f47b201d338 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:02.938892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.962131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:02.962918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:02.963204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:02.963479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:02.973736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:02.974665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:02.981530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:02.982467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:02.983323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 404/801] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:02.984018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.016450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943e330e-4feb-4fd2-9b8b-cc0e4e827f01 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.062177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.063740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.064180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.064451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.071694 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.073316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-461e068c-c4f2-4f11-a5b1-187638174e11 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:03.087539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:03.088421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:03.089351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 398/802] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:03.090130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.090221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.090434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.106712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:03.107891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:03.111345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529639e-84e6-47ce-932c-097a728f101b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.148423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.149110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.149393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.149666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.150634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:03.150942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.152484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.175994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 405/803] 10.222.0.33 () {56 vars in 1149 bytes} [Tue Jul 19 10:52:02 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:03.175994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.177572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 399/804] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:03.178448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.210403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.243558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.280635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cfc614-0419-492a-85ae-795d1d5f4bc9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.281258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9537dea5-4883-4326-9b64-a05b4e5b0254 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.314592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.314740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.315180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.315347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.315456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.315605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.315697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.315881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.316431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:03.316662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:03.316787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.317000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.318086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.318395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.325851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 406/805] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:03.326627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.338379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 400/806] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:03.338379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.421458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.429129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af8c49e-58b7-4989-9ecd-2d1d8f7a95df None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.441362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aac76e5-264b-4dba-8afe-698d06fb0edf None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.462810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.463589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.463874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.464144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.475194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.475316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.475984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.476269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.476542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] PATH_INFO: `/v3/services/network` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.476631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.477217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.478563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1621a880-1a90-4203-9fdb-a89768656082 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:03.490366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:03.491483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:03.492312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 401/807] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/services/network => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:03.493214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975a396-cb8c-431c-b347-eee2b13f16a5 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.493214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975a396-cb8c-431c-b347-eee2b13f16a5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.493345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975a396-cb8c-431c-b347-eee2b13f16a5 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:03.493410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:03.494182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:03.499758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:03.499758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:03.499758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:03.499758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:03.500599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 402/808] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:03.500599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4975a396-cb8c-431c-b347-eee2b13f16a5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.500778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 407/809] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:03.500778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.597668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb06d2e0-b00b-4947-acb9-d8330c9811de None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.599103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4975a396-cb8c-431c-b347-eee2b13f16a5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.630573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.631440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.631725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.631998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/110d74df72174fba99493dedd2e98d08/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.632883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.633665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.633940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=110d74df72174fba99493dedd2e98d08, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.634033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.634211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.635354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.645538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.647022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.647525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 408/810] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:52:03 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/110d74df72174fba99493dedd2e98d08/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:52:03.647996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.657561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-fc460c3e-d1e6-44ae-8f10-4714ca4036bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:03.676789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:03.677764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:03.679468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:03.679468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:03.679468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:03.679468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:03.679468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 403/811] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:03.679770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.680001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.680268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.698682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:03.699920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:03.733426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.778040 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 404/812] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:03.779482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9203ac-6a94-440e-80f5-9b2049442b1c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.785710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.823322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.824029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.824318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.824594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.825356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] RBAC: Adding query filter params (name=network) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:03.825668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.827048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.831760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 409/813] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/services?name=network => generated 124 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:03.832549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.902788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd2451-1aed-4c32-ac3e-118aadba9f9b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.909437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:03.945342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.946340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.946632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.946912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.947525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9abed4-7489-43df-b224-9b24c4931a17 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:03.947636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:03.947945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.949552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.970065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 405/814] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:03.970065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:03.982255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:03.982914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:03.983197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:03.983469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:03.984187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:03.984533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:03.985919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:03.998364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 410/815] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:03.999172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.008714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.089985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de733c06-3c9d-4c7b-b923-fd1ab9698484 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.108344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987b18f5-0f71-4dd5-b683-3eb52c5c35f7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.133013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.134234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.134984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.135704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.136707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.136840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.139013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.144676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.145643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.146067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.146478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.147316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] RBAC: Adding query filter params (type=network) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:04.147759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.149277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:04.152019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:04.153160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:04.154031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 406/816] 10.222.0.33 () {56 vars in 1129 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:04.154955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.161322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 411/817] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:03 2022] GET /identity/v3/services?type=network => generated 350 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:04.162001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.257172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.268921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeae712-d114-4531-b42b-4c20d77838e9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.270296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044660ad-02f6-45b0-8ae7-190a31bf9dcc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.303288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.304466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.304830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.305105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.312216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.314155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.314368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.314937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.315219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.315488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.322753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.324280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:04.329273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:04.330149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:04.331056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:04.333063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 407/818] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:04.334075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:04.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:04.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 412/819] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:04.347047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.400906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.445498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720a04b-1603-4630-a81b-b280fe4ddf1e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.446151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6670e3b9-d4d2-49e4-93d9-3dbb7370098e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.480328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.481003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.481283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.481554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.482281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.483184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.483629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.483837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.484119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.484424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.485155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:04.485459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.487046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.491757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 408/820] 10.222.0.33 () {56 vars in 1124 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:04.492429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.520726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 413/821] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:04 2022] POST /identity/v3/endpoints => generated 327 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:04.520726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.524173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.649321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b645fcc3-e67a-484b-9578-ad409fccb93d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.651809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a008e779-66f6-4c37-9358-5f832a1ba2e7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.659545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.693870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.694633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.694911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.695184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.696017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:04.696274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.697713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.699148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.700306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.700306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.702021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.702021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:04.702183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.704070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.713306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 409/822] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:04.718739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.727368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 414/823] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:04.728402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.827190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:04.851162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc35acbc-950e-47ca-bc59-9536029df5c7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.854688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b544a-8bd7-4826-a712-c8699c6f0561 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:04.887187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.888248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.888536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.888840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2781978834a04fd4a48b4fb2ae2892d7/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.889819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:04.891044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:04.891329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:04.891602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:04.892300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:04.892601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.893991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.903380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 410/824] 10.222.0.33 () {56 vars in 1124 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:04.903380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.910084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=2781978834a04fd4a48b4fb2ae2892d7, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:04.911643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:04.932729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 415/825] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:52:04 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2781978834a04fd4a48b4fb2ae2892d7/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:52:04.932729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:04.948111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:05.050365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb8116-8e4f-4983-9a29-9c84dbc0f076 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:05.054268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600703d3-ae02-4111-b058-6481291dfc17 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:05.095320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:05.097372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.097372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.097372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.097892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:05.099588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:05.099693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:05.100271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.100551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.100854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.114744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:05.136768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 416/826] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:04 2022] POST /identity/v3/endpoints => generated 346 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:05.189190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:05.190833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:05.307278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 411/827] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:04 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:05.313184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:05.400057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:05.441054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753daf05-2db2-495c-bf4f-7748b5a036b0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:05.477385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:05.478176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.478456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.478727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.490500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:05.491927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:05.509385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:05.510274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:05.511165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:05.511165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:05.511165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:05.511165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:05.515159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 417/828] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:05.516635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:05.560322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:05.621804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a373d9f9-407f-46db-8d2e-eee3e4839dd2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:05.663246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:05.663910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.664191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.664463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:05.665525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:05.666880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cf1714-9035-49ae-a921-f4104e45dc5f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:05.681894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 412/829] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:05.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:05.723913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5630d-1e3d-45be-8047-c0b4047713cf None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.724219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5630d-1e3d-45be-8047-c0b4047713cf None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5630d-1e3d-45be-8047-c0b4047713cf None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.726147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 413/830] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:05.731451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.731742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.732027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.750821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:05.752022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:05.801770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9dd7b5-6535-4484-bb93-4fb5711bbef0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:05.824955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:05.827719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 414/831] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:05 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:05.844200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:05.845353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:05.845898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:05.846364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:05.853484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:05.860771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:05.863715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:05.884378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:05.885750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:05.886616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:05.887507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:05.890193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 418/832] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:05.897840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:05.976879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987bf37d-fddd-4f30-a059-54c62d3b8c5f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.023794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.024685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.024995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.025270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.027474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e6f99-3f24-46d6-bdf9-1536d0024951 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.036063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:06.044081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:06.045024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:06.045880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 415/833] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:06.052428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:06.068843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.070039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.070444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.071034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.071891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:06.072296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.074220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.096838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 419/834] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:06.098064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:06.150210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:06.158411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d468c95-8081-4626-995e-4b3bc1fb24ec None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.189115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:06.197829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.198954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.199502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.200006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.201284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:06.201819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.203617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.218506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 416/835] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:06.219863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:06.225540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bb3376-8734-4b03-a42b-48065728cf79 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.269176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.269988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.270271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.270541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.271731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2a1c13591bab47d7ba589e14b83f969e) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:06.272046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.273600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.284780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 420/836] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/role_assignments?user.id=2a1c13591bab47d7ba589e14b83f969e&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:06.344476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9fb59-7b26-48fd-b4cb-1999c15e2e55 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.385436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.386700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.387251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.387726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.400228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.402101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:06.421707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:06.423198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:06.424097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:06.424097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:06.424097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:06.424097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:06.426497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 417/837] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:06.432510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:06.541184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf5ab8f-5ae3-4e1e-b125-3e8f946d7362 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.542875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:06.577230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.577889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.578199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.578473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.579189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:06.579491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.580869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.593108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 421/838] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:06.599190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:06.604207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:06.707852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b46593-6dbc-4ccf-8045-bd0d05963125 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.746018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.746821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.747103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.747377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.755055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.756581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:06.772425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:06.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 418/839] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:06.782330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:06.892464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8187100-6a6e-49b2-a256-bb477aef4041 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:06.929397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:06.930449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:06.930932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:06.931405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:06.932500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:06.933117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:06.934808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c670370-7e3f-4168-964e-c4fb6764a264 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:06.954605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 422/840] 10.222.0.33 () {56 vars in 1135 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:06.960253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.025777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.066892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab1a68b-894c-48ac-8fbb-1ea371ff4780 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.102882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.103537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.103825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.104101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.181895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.183359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.209282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba1d4a2-858e-4402-b7fa-0c7d559f1ae8 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.209569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba1d4a2-858e-4402-b7fa-0c7d559f1ae8 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.209839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba1d4a2-858e-4402-b7fa-0c7d559f1ae8 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.211572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 423/841] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:07.216476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660f1a0-e6d6-4f90-b6c7-5ae8aeaa75ab None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.216795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660f1a0-e6d6-4f90-b6c7-5ae8aeaa75ab None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.217075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660f1a0-e6d6-4f90-b6c7-5ae8aeaa75ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.234595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f660f1a0-e6d6-4f90-b6c7-5ae8aeaa75ab None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:07.235777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f660f1a0-e6d6-4f90-b6c7-5ae8aeaa75ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:07.247027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.252135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.293336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 419/842] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:06 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:07.298646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.321490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 424/843] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:07 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:07.322821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685a8d8-a974-4a26-9c6b-e54398f4669a None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.323112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685a8d8-a974-4a26-9c6b-e54398f4669a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.323391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685a8d8-a974-4a26-9c6b-e54398f4669a None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 425/844] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:07.334467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.334764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.335049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.354719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:07.355910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:07.411259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea1350-62e2-400e-bac2-5d5900f5573d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.427798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 426/845] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:07 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:07.433847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.447498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.448461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.448818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.449097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.460235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.461770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:07.480525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:07.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:07.482406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:07.482406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:07.482406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:07.482406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:07.482406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 420/846] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:07.483142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.501954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.537942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.559593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db722e-65ec-494f-bc10-1db21e1ff27c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.587486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0eea2f-9913-458b-9b78-e94bdac197c5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.597456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.598362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.598645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.598918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.610330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:07.619105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:07.619999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:07.620879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 427/847] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:07.621566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.627811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.628716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.629004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.629278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.629939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.631338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:07.643304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:07.644266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:07.645480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 421/848] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:07.646307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.727480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f553540-8e81-40cf-a1ca-90bc06f08211 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.732881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.755251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d593d9e-02f6-492d-80fb-ab7bf444d26b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.763037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.763709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.763982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.764258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.765012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:07.765318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.767214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.777127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:07.784741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 428/849] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:07.784741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.798907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.799576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.799909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.800193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.800974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:07.801286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.802646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.815329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 422/850] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:07.815329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.889746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6438edd-73b2-417f-a335-61f9bb63dd18 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.926384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43edbec3-5cd1-43bb-9eb7-fb81d9e1e018 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:07.933561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.934374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.934723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.935050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.935773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:07.936205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.937642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.942423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 429/851] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:07.943200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:07.970036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:07.971882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:07.972350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:07.972973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:07.981270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:07.982640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:07.997271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:07.998165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:07.999069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:08.003840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 423/852] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:08.004727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.007555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.011373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.051817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6053143d-d17c-4729-98c9-6925f7255ac8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.088430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.089433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.089546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.089780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.100250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.101710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.111764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab9c57-71cd-4508-8a49-1d30bb75182c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:08.119389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:08.120260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:08.121523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:08.121523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:08.121523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:08.121523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:08.121523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 430/853] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:08.121523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.155026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.155681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.155964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.156320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.156988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:08.157293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.158637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.163224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 424/854] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:08.166115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.235454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b1ad3b-c91d-417e-91cf-067820482d7a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.255326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.274041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.277070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.277933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.278427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.278688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.279610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:08.280027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.282589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.301386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 431/855] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:08.303113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.320612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee3fa46-729b-490e-a922-841066030ad9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.369074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.370262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.370656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.371133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.372398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:08.372936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.374953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.389959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 425/856] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:08.395652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.424519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dd90be-f020-4f4d-94f0-bb5eb32534c3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.469191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.469844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.470126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.470425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.471258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=110d74df72174fba99493dedd2e98d08) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:08.471565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.473016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebeea6a-593e-475b-986e-d35c4c0a4154 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.479938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 432/857] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/role_assignments?user.id=110d74df72174fba99493dedd2e98d08&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:08.481061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ce1bd6-ae99-46d1-b69c-734524a079a3 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.481355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ce1bd6-ae99-46d1-b69c-734524a079a3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.481631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ce1bd6-ae99-46d1-b69c-734524a079a3 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.483004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 433/858] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:08.489429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.489709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.489897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.508275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:08.509623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:08.517578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc0a2e8-0f13-487e-9ac3-55bdd3398328 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.549067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.560285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.574610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.576114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.576891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.578090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.592587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.593898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 434/859] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:08 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:08.601585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.618812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:08.624276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:08.625924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:08.627574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 426/860] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:08.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.743405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162ed01-2dfb-43ed-bf6d-3be433a67de0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.779635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.780472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.780795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.781071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.782304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.783722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:08.799443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:08.800456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:08.801409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:08.802704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 435/861] 10.222.0.33 () {56 vars in 1132 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:08.809126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.847912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:08.890575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea50171f-0ee9-4611-b7bb-7287019efc8b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.926290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545de84-342d-4650-8a9c-07ec1e6a0626 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:08.928391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.929102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.929383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.929654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.930505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:08.930814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.932392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.951964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 427/862] 10.222.0.33 () {56 vars in 1149 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:08.957872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:08.965810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:08.966478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:08.966768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:08.967048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:08.967761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:08.968072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:08.969531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:08.974566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 436/863] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:08.981700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:09.014558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:09.083947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3ab5f9-a5f0-44b4-b413-fb54656eaa45 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:09.090060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:09.095611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb287a3f-3165-47a4-8ba7-8ba92b6bc442 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:09.119489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:09.120135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:09.120420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:09.120713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:09.134160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:09.134820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:09.135107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:09.135381 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:09.136186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:09.136495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:09.138011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:09.143322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 437/864] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:09.195998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:09.197503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:09.247847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:09.308065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 428/865] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:08 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 19 10:52:09.314182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:09.421055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e078b2-f9d6-4e59-afe8-5db70d2b9e28 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:09.460298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:09.461119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:09.461404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:09.461683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:09.473747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:09.475195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:09.493054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:09.493962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:09.495274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:09.495274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:09.495274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:09.495274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:09.495274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 438/866] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:09.500145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:09.605556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd501db2-9d66-43ac-abda-3dd75c6f3e3c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:09.642746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:09.643743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:09.644221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:09.644735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:09.645671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:09.646167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:09.647716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:09.661082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 429/867] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:09.666226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:09.686298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:09.783705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce4b024-e46f-450d-9481-90aba5ce49f2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:09.819653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:09.820814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:09.821365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:09.821851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:09.829061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:09.830703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:09.845947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:09.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:09.847719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:09.850133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 439/868] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:09.856110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:09.862266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:09.974657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d00509-a00d-4ffd-9bfc-4a0a53968f0e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:10.011247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:10.011913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.012200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.012477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.013298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:10.013609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:10.015029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f32c1cd-c347-40a1-b0e6-3e00943de8a0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:10.028650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 430/869] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:10.034600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:10.114805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:10.142073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d759bd7d-11a6-42d6-aa4a-93424c93adde None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:10.177964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:10.178667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.178948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.179271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.180107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2781978834a04fd4a48b4fb2ae2892d7) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:10.180419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:10.181876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc3d709-1ab2-4250-97b4-7ad0cb3b79f3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:10.188891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 440/870] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:10 2022] GET /identity/v3/role_assignments?user.id=2781978834a04fd4a48b4fb2ae2892d7&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:10.296960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35d20ac-0424-4bc5-9689-508763e4e380 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.297607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35d20ac-0424-4bc5-9689-508763e4e380 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.298090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35d20ac-0424-4bc5-9689-508763e4e380 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.300031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 431/871] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:10.306050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c47e0a-1290-4cf3-8f54-8cda78e7bb4e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.306701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c47e0a-1290-4cf3-8f54-8cda78e7bb4e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.307203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c47e0a-1290-4cf3-8f54-8cda78e7bb4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.326010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-66c47e0a-1290-4cf3-8f54-8cda78e7bb4e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:10.327610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-66c47e0a-1290-4cf3-8f54-8cda78e7bb4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:10.404895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 441/872] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:10 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:10.427513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35d20ac-0424-4bc5-9689-508763e4e380 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:10.499464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:10.545341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35d20ac-0424-4bc5-9689-508763e4e380 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:10.581515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89a8912-579b-4d10-88ac-3f5dd8eb4f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:10.582167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a8912-579b-4d10-88ac-3f5dd8eb4f5a None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.582452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a8912-579b-4d10-88ac-3f5dd8eb4f5a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.582722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a8912-579b-4d10-88ac-3f5dd8eb4f5a None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.583443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89a8912-579b-4d10-88ac-3f5dd8eb4f5a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:10.584886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89a8912-579b-4d10-88ac-3f5dd8eb4f5a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:10.599233 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 432/873] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:10 2022] POST /identity/v3/services => generated 251 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:10.671370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b62e6d-b45e-4028-8ac9-1e722b351eb3 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.671656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b62e6d-b45e-4028-8ac9-1e722b351eb3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.672410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b62e6d-b45e-4028-8ac9-1e722b351eb3 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.674146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 442/874] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:10.679811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.680125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.680367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.698932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:10.700265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:10.781056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 433/875] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:10 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:10.805961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b62e6d-b45e-4028-8ac9-1e722b351eb3 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:10.881708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:10.918101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b62e6d-b45e-4028-8ac9-1e722b351eb3 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:10.954220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:10.955093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:10.955375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:10.955649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:10.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:10.957765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:10.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:10.971648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:10.972479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:10.976798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 443/876] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:10.978346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:11.032049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:11.083041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ecd495-6bf5-42d2-b2ca-992859c839fe None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:11.118784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:11.119455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:11.119786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:11.120061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:11.120781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:11.121085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:11.122437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25e0bb-7fe4-4c2e-bce7-8c269897f86f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:11.131834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 434/877] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:11.133219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:11.235139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3537d8-87bb-47d3-9e48-54c0da774766 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:11.263702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:11.269298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:11.269970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:11.270526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:11.270799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:11.271518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:11.271821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:11.273224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0775297a-1d54-4333-be18-66fa0024669e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:11.282611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 444/878] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:11 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:11.328801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98cc6c8-2d84-49c3-998d-1d7d4a2f3902 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:11.329122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98cc6c8-2d84-49c3-998d-1d7d4a2f3902 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:11.329363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98cc6c8-2d84-49c3-998d-1d7d4a2f3902 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:11.331101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 435/879] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:11.336958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962c2d2-1b8e-4988-93bf-3122c9ca2cf2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:11.337263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962c2d2-1b8e-4988-93bf-3122c9ca2cf2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:11.337509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962c2d2-1b8e-4988-93bf-3122c9ca2cf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:11.356422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b962c2d2-1b8e-4988-93bf-3122c9ca2cf2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:11.357685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b962c2d2-1b8e-4988-93bf-3122c9ca2cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:11.436018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 445/880] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:11 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:11.458914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98cc6c8-2d84-49c3-998d-1d7d4a2f3902 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:11.564006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98cc6c8-2d84-49c3-998d-1d7d4a2f3902 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:11.578552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:11.599163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5cf46-e294-4cf2-be0b-50846046f26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:11.599821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5cf46-e294-4cf2-be0b-50846046f26e None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:11.600103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5cf46-e294-4cf2-be0b-50846046f26e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:11.600375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5cf46-e294-4cf2-be0b-50846046f26e None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:11.601179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5cf46-e294-4cf2-be0b-50846046f26e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:11.602531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5cf46-e294-4cf2-be0b-50846046f26e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:11.624829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 436/881] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:11 2022] POST /identity/v3/services => generated 269 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:12.422395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578ae8d-31f4-4f55-b475-fe8c298b162d None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:12.423407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578ae8d-31f4-4f55-b475-fe8c298b162d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:12.423894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578ae8d-31f4-4f55-b475-fe8c298b162d None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:12.425788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 446/882] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:12.431270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:12.431784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:12.432292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:12.450186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:12.451668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:12.526693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 437/883] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:12 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:12.551373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3578ae8d-31f4-4f55-b475-fe8c298b162d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:12.655251 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3578ae8d-31f4-4f55-b475-fe8c298b162d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:12.685299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:12.690145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9757ad7d-a63f-49dc-9e86-5203b365ce5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:12.690956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9757ad7d-a63f-49dc-9e86-5203b365ce5f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:12.691227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9757ad7d-a63f-49dc-9e86-5203b365ce5f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:12.691499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9757ad7d-a63f-49dc-9e86-5203b365ce5f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:12.702408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9757ad7d-a63f-49dc-9e86-5203b365ce5f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9757ad7d-a63f-49dc-9e86-5203b365ce5f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:12.710510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:12.711461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:12.712301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 447/884] 10.222.0.33 () {56 vars in 1120 bytes} [Tue Jul 19 10:52:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:12.720024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:12.728858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df813f4a-66df-4195-a036-72d153a805b7 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:12.729161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df813f4a-66df-4195-a036-72d153a805b7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:12.729443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df813f4a-66df-4195-a036-72d153a805b7 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:12.731046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 448/885] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:12.736416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:12.736727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:12.737291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:12.754398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:12.755594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:12.834181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 449/886] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:12 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:12.842157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e1b8d-13db-46d5-afa9-24e49020cbf9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:12.844258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:12.857686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:12.885368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:12.886048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:12.886327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:12.886599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:12.887454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:12.887856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:12.889252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:12.898924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 438/887] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:52:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:12.905806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:12.982343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c72dd34-6d10-4bad-8227-b3b1603396c3 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.007613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.034151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.034947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.035227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.035497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.036146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.037543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.042395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95de782-085c-4629-9ff7-060f5cd8b706 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:13.049414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:13.050459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:13.051284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 450/888] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:12 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:13.058195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.087977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.089113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.089603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.090117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.102009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.103777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:13.121657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:13.122554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:13.123438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:13.123438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:13.123438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:13.123438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:13.125808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 439/889] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:13.130980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.148354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.180444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596799fb-f498-4a2a-9df2-8b675e3d7d15 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.226160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.227608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.228351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.228865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.229975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:13.230797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.232866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.243925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d199d08-be24-486b-989d-ea80266e1799 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.244622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 451/890] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:13.246604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.279780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.280494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.280814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.281346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.282036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:13.282348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.283728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.298208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.301256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 440/891] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:13.302328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.357462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48097166-4617-45d0-bd3e-611c78bfdffd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.382659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.406242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.407627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.408366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.408861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.409771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:13.409965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bada4d-7906-47cc-b7d5-a4ec0cc3d869 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.410438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.412938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc14270-75f6-4f64-b5e4-64ababe7c4cb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.418070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 452/892] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:13.418070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64b079f-a8cd-4c33-a9c1-a89f1265c923 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.418070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64b079f-a8cd-4c33-a9c1-a89f1265c923 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.418070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64b079f-a8cd-4c33-a9c1-a89f1265c923 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.418070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 453/893] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:13.423857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64b079f-a8cd-4c33-a9c1-a89f1265c923 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.456188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.457550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.458272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.458961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.466995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.468435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-cd7177e0-08d0-415b-8019-8a818ea88d71 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:13.483787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:13.484830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:13.485788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 441/894] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:13.486394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.487594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.504839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:13.506049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:13.535347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64b079f-a8cd-4c33-a9c1-a89f1265c923 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.544385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.572268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.572959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.573402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.573684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.574472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] RBAC: Adding query filter params (interface=public, service_id=18f7a705877a4e8e8236285eeaec3ce1, region_id=RegionOne) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:13.574784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.576193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.577863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 442/895] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:13 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:13.589160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 454/896] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/endpoints?service_id=18f7a705877a4e8e8236285eeaec3ce1&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:13.589309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.591436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.665448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.702413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89ac4b-29ae-444e-8b6d-b1a2ad133402 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.710949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae6fbd-2dec-424a-a39b-0bab06a52f8d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.737982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.738578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.738858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.739132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.739817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.741212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.756726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 455/897] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/services => generated 1268 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 10:52:13.756726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.757015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.757015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.757015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.757015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.757358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:13.757358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.757358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be710724-3027-4852-a9ca-5a8fe75869fd None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.776068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 443/898] 10.222.0.33 () {56 vars in 1149 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:13.777489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a93d6-3e63-4e03-ae04-54c3791158a4 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.777843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a93d6-3e63-4e03-ae04-54c3791158a4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.778126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a93d6-3e63-4e03-ae04-54c3791158a4 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.779647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 444/899] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:13.781940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a93d6-3e63-4e03-ae04-54c3791158a4 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:13.793044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.866581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8763dee-89ac-4fe8-85fe-6f8c03a389b4 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.906283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee02f7eb-476e-462d-87a1-29a39341925f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.906720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a93d6-3e63-4e03-ae04-54c3791158a4 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:13.906942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee02f7eb-476e-462d-87a1-29a39341925f None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.907787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee02f7eb-476e-462d-87a1-29a39341925f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.907787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee02f7eb-476e-462d-87a1-29a39341925f None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.908203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee02f7eb-476e-462d-87a1-29a39341925f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:13.909644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee02f7eb-476e-462d-87a1-29a39341925f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:13.925915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 456/900] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:13 2022] POST /identity/v3/services => generated 241 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:13.927071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.927362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.927636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:13.936318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:13.951865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:13.956822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:13.971507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:13.972809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:13.973308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:13.974319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.065004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 457/901] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:13 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:14.068583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.070157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.087393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.123305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.188192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 445/902] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:13 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:14.197170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.220731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82325b6c-09c4-4220-987d-d5dde181f10f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.225065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.259118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.259930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.260212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.260483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.261164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.262558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:14.275138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:14.276096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:14.276965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 458/903] 10.222.0.33 () {56 vars in 1132 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:14.282914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.335706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c0616d-97d0-4b12-8a29-9c233eeafc31 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.381245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.383924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.385145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.385975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.386746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.392629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ef0fa7-0c27-493c-ab18-8db1ff461cd9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.401358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.403998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:14.427153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:14.428120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:14.429020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:14.429020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:14.429020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:14.429020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:14.433057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 446/904] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:14.433920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.436109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.437489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.437828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.438114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.438832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:14.439911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.440504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.446361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 459/905] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:14.451886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.522304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.558256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1672b49-b53d-48b1-aba6-e5ea04a67cdf None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.582293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e70d95-7cf9-455e-a50e-ba61e98529ab None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.598987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.600922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.601455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.602189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.603392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:14.604286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.605908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.622152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.622813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.623168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.623639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 447/906] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:14.627927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.628675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:14.629016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.629374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.630533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.641400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 460/907] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:14.642666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.663604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.752912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86daeb5c-1e67-4877-b98b-4ef428ddb69c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.770768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37352ff7-1281-47ee-8042-33b9c375697d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.789097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.789831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.790190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.790747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] PATH_INFO: `/v3/endpoints` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.791387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] RBAC: Adding query filter params (interface=public, service_id=3fa447bfb4b649c7a370864a0331488e, region_id=RegionOne) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:14.791769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.793258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.803304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 461/908] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/endpoints?service_id=3fa447bfb4b649c7a370864a0331488e&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:14.804205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.809643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.818567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.819767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.820342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.820843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.830341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.832423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:14.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:14.850827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:14.851734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:14.856055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 448/909] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:14.857138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:14.925074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:14.936947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e82a5c-29fb-4656-ae06-1a4d54c4e41b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.977563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:14.978376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:14.978763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:14.979157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:14.979930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:14.981427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:14.990924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440626a4-08ac-4660-b53d-293f271cf0b5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:14.993207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 462/910] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/services => generated 1497 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 10:52:15.033865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:15.034808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:15.035275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:15.035701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:15.036672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:15.037168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:15.038825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7bd72-ccb6-4915-92cc-57290b38f507 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:15.053701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 449/911] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:15.060068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:15.175495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd85c384-b6c9-40dc-ae87-fc52d17deb85 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:15.211581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:15.212224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:15.212505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:15.212807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:15.213633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] RBAC: Adding query filter params (role.id=4a90181c3f7847a7b2f8472e19bdd196, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2781978834a04fd4a48b4fb2ae2892d7) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:15.213941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:15.215347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe673afe-5107-441d-9ec8-083d576a78b7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:15.221911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 463/912] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:15 2022] GET /identity/v3/role_assignments?user.id=2781978834a04fd4a48b4fb2ae2892d7&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=4a90181c3f7847a7b2f8472e19bdd196 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:15.847999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e9c7f8-9676-4844-99d3-20420a3a7b84 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:15.848522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e9c7f8-9676-4844-99d3-20420a3a7b84 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:15.848600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e9c7f8-9676-4844-99d3-20420a3a7b84 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:15.850196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 450/913] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:15.855599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b5ddeb-9391-49d3-82ed-373de06f8da1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:15.855994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b5ddeb-9391-49d3-82ed-373de06f8da1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:15.856158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b5ddeb-9391-49d3-82ed-373de06f8da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:15.873346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-88b5ddeb-9391-49d3-82ed-373de06f8da1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:15.874519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-88b5ddeb-9391-49d3-82ed-373de06f8da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:15.877906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:15.948802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 464/914] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:15 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:15.964921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e9c7f8-9676-4844-99d3-20420a3a7b84 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.017634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:16.072982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e9c7f8-9676-4844-99d3-20420a3a7b84 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.074508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa084b8-a2bd-4204-8896-cc5c05d8a119 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.074818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa084b8-a2bd-4204-8896-cc5c05d8a119 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.075073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa084b8-a2bd-4204-8896-cc5c05d8a119 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.080353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 465/915] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:16.081837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.082354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.082827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.100301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:16.101769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:16.113084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.113883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.114167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.114444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.115427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.117106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:16.132821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:16.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:16.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 451/916] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:15 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:16.140028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.188828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 466/917] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:16 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:16.205305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.255366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506d0ea0-2439-4c88-97a6-0b2e25c36704 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.274507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:16.282251 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:16.304227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.304929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.305216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.305493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.306263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:16.306567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.308508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.316776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 452/918] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:16.321975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.325630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d9e109-db53-40fd-a224-87baa91f6a11 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.366486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.367278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.367473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.367741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.368395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.369807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:16.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:16.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:16.384776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:16.389041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 467/919] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/services/object-store => generated 93 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:16.390596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.459682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e56fad-0691-4d67-95af-eec40ddeaf99 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.501037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be0cf5-0d48-4bf9-8901-e3a83c47c07f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.507165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.507825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.508106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.508373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.509102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:16.509402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.510901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.516748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 453/920] 10.222.0.33 () {56 vars in 1114 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:16.522653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.545171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.546072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.546527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.547028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.547898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:16.548375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.549954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.551338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:16.560770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 468/921] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:16.563472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.580223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:16.655879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5b92c3-9e68-424e-88f8-2c907b0efb42 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.688622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e2674-2d7f-45e8-ab11-e55b82639db2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.695302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.696813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.697481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.698180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.699673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.701666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.725618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 454/922] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:16 2022] POST /identity/v3/endpoints => generated 346 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:52:16.737621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.738662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.739187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.739671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.740641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:16.741261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.743110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.752829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 469/923] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:16.759502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:16.886516 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:16.888558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef985e7-a698-4b8b-b557-7a52648a20c0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:16.938608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:16.939287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:16.939570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:16.939843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:16.940638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] RBAC: Adding query filter params (interface=public, service_id=c145b407133147b08c560b609929a2bb, region_id=RegionOne) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:16.940976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:16.942335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fd1968-9a79-44b4-8470-009d561fc312 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:16.952304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 455/924] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/endpoints?service_id=c145b407133147b08c560b609929a2bb&interface=public®ion_id=RegionOne => generated 194 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:16.954363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.019060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:17.062916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca843d00-e519-4378-aebd-72f101864dc2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.098499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580827b-7c6a-4f6e-9928-df94ed650126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.099010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580827b-7c6a-4f6e-9928-df94ed650126 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.099389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580827b-7c6a-4f6e-9928-df94ed650126 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.099762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580827b-7c6a-4f6e-9928-df94ed650126 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.100550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580827b-7c6a-4f6e-9928-df94ed650126 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:17.102097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580827b-7c6a-4f6e-9928-df94ed650126 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:17.108656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 470/925] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:16 2022] GET /identity/v3/services => generated 1497 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 10:52:17.163567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3a288-71fc-4e80-95ad-8cd699628a79 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.163858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3a288-71fc-4e80-95ad-8cd699628a79 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.164146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3a288-71fc-4e80-95ad-8cd699628a79 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.165908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 456/926] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:17.171845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6704e6-6f3d-4781-b73a-0d8b92301cd0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.172147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6704e6-6f3d-4781-b73a-0d8b92301cd0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.172434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6704e6-6f3d-4781-b73a-0d8b92301cd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.190846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-3d6704e6-6f3d-4781-b73a-0d8b92301cd0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:17.191983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3d6704e6-6f3d-4781-b73a-0d8b92301cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:17.271005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 471/927] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:17 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:17.278434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abeda64-ac8d-4d12-b0e6-9fda98a9e5fa None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.278731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abeda64-ac8d-4d12-b0e6-9fda98a9e5fa None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.278996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abeda64-ac8d-4d12-b0e6-9fda98a9e5fa None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.280613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 457/928] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:17.285731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.286023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.286301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.295695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abeda64-ac8d-4d12-b0e6-9fda98a9e5fa None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.305671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:17.306866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:17.365886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:17.388521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 472/929] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:17 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:17.410641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.416676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abeda64-ac8d-4d12-b0e6-9fda98a9e5fa None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.454034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.454768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.455054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.455328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.455996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:17.457392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:17.470537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:17.471523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:17.472367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 458/930] 10.222.0.33 () {56 vars in 1132 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:17.481774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.513460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f961e0a7-1bda-4811-9b2b-8c68a896a947 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.523239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:17.551169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.551970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.552250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.552523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.563701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:17.571853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:17.572820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:17.573661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:17.577994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 473/931] 10.222.0.33 () {56 vars in 1120 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:17.578890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.614807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f543a7-704e-4ed4-ae8c-a9d311b5aef6 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.627340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:17.666897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.667779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.668065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.668586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.669793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:17.670106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:17.672531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:17.679087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 459/932] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:17.684510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.693392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4049cb12-7d50-420e-b4b1-8244901aa11c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.731322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.732021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.732366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.732719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.733541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:17.733913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:17.736055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:17.749863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 474/933] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:17.750867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.798959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:17.812523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f91d97-faa9-49b3-8548-703feb3a74ee None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.853909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.854801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.855267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.855710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.856606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:17.857163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:17.858867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:17.867806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 460/934] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:17.872618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce196150-5de4-4b73-9ebb-b738f68abbf1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:17.877161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:17.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:17.922589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:17.923470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:17.923823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:17.924168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:17.939054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:17.940931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:17.965836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:17.967353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:17.968373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:17.968373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:17.968373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:17.968373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:17.972783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 475/935] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:17.977765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:18.006394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81707894-5ee6-4ad8-a03a-727e26c4b3b7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:18.044716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:18.045736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.046213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.046686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.047648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:18.049242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:18.073092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 461/936] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:17 2022] POST /identity/v3/endpoints => generated 347 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:18.116042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebe9662-86db-4707-b50f-2f31c0470b50 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:18.162612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:18.163282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.163567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.163844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.164568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:18.164907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:18.166350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:18.184151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 476/937] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:18.186915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:18.203608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:18.298593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b90145-ba57-4508-8358-bb4fcb5f6f33 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:18.333934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:18.334751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.335036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.335312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.343259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:18.345304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:18.361167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:18.362479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:18.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 462/938] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:52:18 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:18.370558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:18.386564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:18.480879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee5feda-3f6d-405e-8285-f85412bfd4d2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:18.518341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:18.519380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.519866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.520352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.521419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:18.521931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:18.523550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:18.541850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 477/939] 10.222.0.33 () {56 vars in 1149 bytes} [Tue Jul 19 10:52:18 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:18.547773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:18.625794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:18.658331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10b81ad-0bed-430b-b81d-731993fc0e56 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:18.693249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:18.694040 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.694327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.694606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.706064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:18.707525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-8fb58f68-b381-4d46-9faf-15ecc5dcba0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:18.724861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:18.725787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:18.726669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:18.726669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:18.726669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:18.726669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:18.726669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 463/940] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:18.733171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:18.808987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:18.842895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a062c69b-b9f5-41db-8727-9f31f1d08625 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:18.880574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:18.881862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.882654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.883431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.885209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:18.886084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:18.889001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:18.898048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c667c15f-f5d2-4a27-ae0b-53e078455069 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.898364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c667c15f-f5d2-4a27-ae0b-53e078455069 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.898613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c667c15f-f5d2-4a27-ae0b-53e078455069 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.900245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 464/941] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:18.903538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 478/942] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:18.905952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:18.906461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:18.906891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:18.909144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:18.926987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:18.928529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:19.027294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6824dacb-4a6c-4ee4-8963-34fc8c4c2878 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.027915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 465/943] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:18 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:19.051890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.059450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:19.067109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.067912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.068192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.068468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.076061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.077579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:19.093640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:19.094610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:19.095742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 479/944] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:19.100872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.116623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:19.176350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f72939f-767a-4013-89d7-91ff81b7a051 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.219937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.220950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.221321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.222024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.223123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.224845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.226914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING keystone.server.flask.common [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 19 10:52:19.231467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06297ce-61f1-4572-a759-473240cb2005 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.249603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.sql.core [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 19 10:52:19.249603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 19 10:52:19.249603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [parameters: {'id': '4ad784d9534e43e99305b1784d2e74b8', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 19 10:52:19.249603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80152) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4ad784d9534e43e99305b1784d2e74b8.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4ad784d9534e43e99305b1784d2e74b8. Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result = self._query(query) Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application conn.query(q) Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 19 10:52:19.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result.read() Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 19 10:52:19.259622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application next(self.gen) Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application yield resource Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application next(self.gen) Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.session.rollback() Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.force_reraise() Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise self.value Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application session.commit() Jul 19 10:52:19.260627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._prepare_impl() Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.session.flush() Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._flush(objects) Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application compat.raise_( Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application flush_context.execute() Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 19 10:52:19.261556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rec.execute(self) Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 19 10:52:19.262451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result = self._query(query) Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application conn.query(q) Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application result.read() Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 19 10:52:19.263249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [parameters: {'id': '4ad784d9534e43e99305b1784d2e74b8', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:19.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4ad784d9534e43e99305b1784d2e74b8. Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.265922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 466/945] 10.222.0.33 () {60 vars in 1140 bytes} [Tue Jul 19 10:52:19 2022] POST /identity/v3/projects => generated 262 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:19.266480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.286182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.287320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.287865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.288349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.289521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:19.290146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.291985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.307022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 480/946] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:19.312580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.367687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:19.384389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578cfb1c-78d8-4cca-996e-54815579ea1f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.394147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:19.423002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e2ac5-888e-4154-a656-a8043beb7c15 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.423470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.424284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.424641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.424945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.432136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.433854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-e6d067e6-8ac2-4793-9a24-9ccad398bcee None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:19.450027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:19.450888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:19.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 467/947] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:19.454615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede3da9-0778-4651-b6ae-dc4d09e9cb95 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.454898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede3da9-0778-4651-b6ae-dc4d09e9cb95 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.455167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede3da9-0778-4651-b6ae-dc4d09e9cb95 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.459221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 468/948] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:19.459925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ede3da9-0778-4651-b6ae-dc4d09e9cb95 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.465612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41596dc8-28b7-49fd-925d-095b57072a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.466497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41596dc8-28b7-49fd-925d-095b57072a18 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.466777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41596dc8-28b7-49fd-925d-095b57072a18 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.467049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41596dc8-28b7-49fd-925d-095b57072a18 None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2781978834a04fd4a48b4fb2ae2892d7/roles/4a90181c3f7847a7b2f8472e19bdd196` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.469751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41596dc8-28b7-49fd-925d-095b57072a18 None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=2781978834a04fd4a48b4fb2ae2892d7, role_id=4a90181c3f7847a7b2f8472e19bdd196)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.471546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41596dc8-28b7-49fd-925d-095b57072a18 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.485740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 481/949] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:52:19 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2781978834a04fd4a48b4fb2ae2892d7/roles/4a90181c3f7847a7b2f8472e19bdd196 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 10:52:19.486717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.487004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.487535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.507789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:19.509064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:19.596179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ede3da9-0778-4651-b6ae-dc4d09e9cb95 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.626034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 482/950] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:19 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:19.642467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.643202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.643539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.643819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.644603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:19.644952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.646407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.647793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.676809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 469/951] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:19 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:19.705180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:19.773407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83d5df1-04d7-4403-ba2d-9e9f788d6e32 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.811442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.812253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.813193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.813467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.814176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.815567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:19.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:19.829681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:19.830523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 483/952] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:19 2022] GET /identity/v3/services/object-store => generated 93 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:19.836342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:19.867297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:19.947957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88997376-1c54-4b1b-839b-1aa0f42c81ec None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:19.983300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:19.983887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:19.984169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:19.985090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:19.985807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:19.986111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:19.987740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:19.993406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 470/953] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:19.998783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:20.106227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb089a30-af91-4e58-8a6d-1f8a9f24ff64 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:20.108573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:20.142763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:20.143431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.143713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.143986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.144719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:20.145084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:20.146513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f3448-1e0d-4eb3-824b-02fee995d523 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:20.156703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 484/954] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:20.158242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:20.233164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500648a3-a043-4ce7-a130-9181266aa31d None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.233548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500648a3-a043-4ce7-a130-9181266aa31d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.233715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500648a3-a043-4ce7-a130-9181266aa31d None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.235965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 485/955] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:20.241849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.242377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.242748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.261714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:20.266673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:20.267889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:20.275061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d656e2-bb41-4265-b1a5-a20d4910f087 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:20.319121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:20.320163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.320606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.321228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.322540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:20.324355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:20.352769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 486/957] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:20 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:20.353094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 471/957] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:20 2022] POST /identity/v3/endpoints => generated 349 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:20.370597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:20.482148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:20.486371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd52159-99b5-404d-9ffe-f3abfab31c51 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:20.529773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:20.530570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.530849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.531121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.532395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:20.533918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:20.545750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:20.546881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:20.547760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 487/958] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:20.553176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:20.661277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e047a98-e775-4bb7-891d-3dd5729ff7bb None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:20.698308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:20.698996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.699286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.699565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.700294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] RBAC: Adding query filter params (name=compute) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:20.700603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:20.702028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f143977-32b6-471e-85f7-2ff11282eaaf None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:20.708982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 472/959] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:20.714262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:20.822818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3a5c1c-1c42-4102-b44a-b406234e85b3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:20.858240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:20.858902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:20.859189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:20.859499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:20.860227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] RBAC: Adding query filter params (type=compute) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:20.860513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:20.861912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9154d0ec-4492-4533-9920-532db5579acb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:20.867249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 488/960] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:20 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:21.641056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d2e089-00d4-4262-a365-cd8715455a36 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:21.641056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d2e089-00d4-4262-a365-cd8715455a36 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:21.641056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d2e089-00d4-4262-a365-cd8715455a36 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:21.641056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 473/961] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:21.645777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d12070-1d37-456c-a696-46624a81ea9f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:21.646246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d12070-1d37-456c-a696-46624a81ea9f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:21.646400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d12070-1d37-456c-a696-46624a81ea9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:21.663892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-96d12070-1d37-456c-a696-46624a81ea9f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:21.665108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-96d12070-1d37-456c-a696-46624a81ea9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:21.682388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:21.738681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 489/962] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:21 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:21.761414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d2e089-00d4-4262-a365-cd8715455a36 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:21.776848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:21.867565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d2e089-00d4-4262-a365-cd8715455a36 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:21.894294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494e113-20f8-4959-92a8-f6fdf07f4036 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:21.894888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494e113-20f8-4959-92a8-f6fdf07f4036 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:21.895367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494e113-20f8-4959-92a8-f6fdf07f4036 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:21.897203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 490/963] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:21.902270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:21.902873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:21.903397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:21.904280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:21.905119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:21.905402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:21.905675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:21.918500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:21.922444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:21.923853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:21.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:21.927616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:21.928454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 474/964] 10.222.0.33 () {56 vars in 1120 bytes} [Tue Jul 19 10:52:21 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:21.933967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.016060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 491/965] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:21 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:22.016289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.036522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.052132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e68579-d2cb-4894-84fe-bf9fc2ac1330 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.091073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.091844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.092134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.092410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.093300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:22.093605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.095693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.111698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 475/966] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:52:21 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:22.113147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.129274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.149457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f391a0-1c5d-4ca9-8b41-5998a10508e4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.184722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.185400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.185688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.185965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.186670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.188140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.189491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING keystone.server.flask.common [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 19 10:52:22.202764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.sql.core [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 19 10:52:22.202764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 19 10:52:22.202764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [parameters: {'user_id': 'b8f269ea42d54f219626ae07b2fe4805', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 19 10:52:22.202764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80153) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 19 10:52:22.224559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a30e081-5372-4610-8110-74bd862b66d4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application result = self._query(query) Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application conn.query(q) Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application result.read() Jul 19 10:52:22.225817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 19 10:52:22.226649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application next(self.gen) Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application yield resource Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application next(self.gen) Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self.session.rollback() Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self.force_reraise() Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise self.value Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application session.commit() Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 19 10:52:22.227625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._prepare_impl() Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self.session.flush() Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._flush(objects) Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application compat.raise_( Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application flush_context.execute() Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rec.execute(self) Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 19 10:52:22.228525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 19 10:52:22.230936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application result = self._query(query) Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application conn.query(q) Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application result.read() Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 19 10:52:22.232536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [parameters: {'user_id': 'b8f269ea42d54f219626ae07b2fe4805', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:22.233778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 19 10:52:22.241764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 492/967] 10.222.0.33 () {60 vars in 1131 bytes} [Tue Jul 19 10:52:22 2022] POST /identity/v3/users => generated 160 bytes in 192 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:22.243043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.266678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.267474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.267762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.268037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.281105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.282636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.284233 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:22.314485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:22.315436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:22.316792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:22.316792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:22.316792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:22.316792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:22.316792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 476/968] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:22.321449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.372255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe74bfa-216c-4578-8ff3-04cd39b1e480 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.423219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.424858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.425767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.426572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.437369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.440787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.448273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5613b1-db72-455e-b939-424c37014005 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:22.461320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:22.462282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 19 10:52:22.463227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:22.466705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 493/969] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/users/cinder => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:22.472264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.477702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.494024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.495139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.495431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.495708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.496916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:22.497027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.498379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08816b91-889b-4ce3-af92-2ececcf25084 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.516259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 477/970] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:22.516259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f929d-71e8-4356-bff4-6f11f3e9c66a None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.516259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f929d-71e8-4356-bff4-6f11f3e9c66a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.516259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f929d-71e8-4356-bff4-6f11f3e9c66a None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.516259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 478/971] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:22.518058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216f929d-71e8-4356-bff4-6f11f3e9c66a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.571337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.593382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08847df-d919-4184-99f1-ea116cf63b9a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.633174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.633894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.634186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.634563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.635377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:22.635687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.637139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656d445f-31f5-4a9b-a5d7-003d900d36be None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.641920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216f929d-71e8-4356-bff4-6f11f3e9c66a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.676960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 494/972] 10.222.0.33 () {56 vars in 1101 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:22.679157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.680223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.680726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.681194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.686431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.688657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.689040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.692807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.694651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.710018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:22.711598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:22.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:22.713463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 19 10:52:22.714404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:22.717193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 479/973] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:22.726004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.783244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.800295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 495/974] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:22 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:22.823124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.860755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9c420e-17a9-4648-b9e3-c1ff1c7efda5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.873120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:22.916858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.917540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.917824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.918096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:22.919679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.921481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:22.951979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d854e-b6dd-4d73-a15a-120860b78f4e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:22.952186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 480/975] 10.222.0.33 () {56 vars in 1149 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:22.957165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:22.995574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:22.996482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:22.996957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:22.997206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:22.997860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:22.999235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f715ecfd-a81e-410b-8fde-46775a7db68f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:23.011945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:23.013216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:23.014147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 496/976] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/services/object-store => generated 93 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:23.016899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc1b5f3-ee17-49e2-a649-89c67ac44fae None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.017191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc1b5f3-ee17-49e2-a649-89c67ac44fae None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.017479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc1b5f3-ee17-49e2-a649-89c67ac44fae None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.018802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 497/977] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:23.019496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc1b5f3-ee17-49e2-a649-89c67ac44fae None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.069532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b838d3a-5801-480e-8832-bdc1e6a930db None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.075718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.104320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.104993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.105278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.105560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.121288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc1b5f3-ee17-49e2-a649-89c67ac44fae None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.135881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.154597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.155249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.155536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.155811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.157043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:23.157352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.158708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420afb87-2335-486a-9d13-62f60fedcda4 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.164066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 498/978] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:23.165291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.165575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.165854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.174504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.175940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.184051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:23.185266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:23.250695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 499/979] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:23 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:23.251416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.273775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 481/980] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:22 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:23.274455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.302278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.338365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.351504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef1cecf-92c8-445a-9b71-6511570b5aed None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.376085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d3923e-d86d-413c-b1d9-2f88fddedec7 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.384432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.385121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.385402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.385674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.386372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:23.386674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.388027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.393198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 500/981] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:23.393954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.410259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.410932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.411212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.411483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.412195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.413574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.426358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 482/982] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:23 2022] POST /identity/v3/services => generated 242 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:23.427183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.507766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459debcc-3d04-4d29-8336-907a6b8b58fb None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.524219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.549485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8262d75-193c-4fa8-9511-94c7fa9f0743 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.554706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.555514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.555796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.556068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.566920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.568370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.573436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:23.589073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:23.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:23.590944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:23.590944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:23.590944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:23.590944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:23.590944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 501/983] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:23.596495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.596797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.597641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.597923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.598194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.599002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] RBAC: Adding query filter params (interface=admin, service_id=c145b407133147b08c560b609929a2bb, region_id=RegionOne) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:23.599489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.601740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.607707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 483/984] 10.222.0.33 () {56 vars in 1249 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/endpoints?service_id=c145b407133147b08c560b609929a2bb&interface=admin®ion_id=RegionOne => generated 193 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:23.613343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.701238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9d09f-48d4-40f6-bb38-cecaba2f4e74 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.717180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4dbbd9-8d62-4728-8301-597de7d2c703 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.736123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.736804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.737086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.737611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.738322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:23.738623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.740011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.752120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 502/985] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:23.757531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.757709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:23.758260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.758478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.758750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.759435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.760828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.768049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 484/986] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/services => generated 1727 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 10:52:23.784012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:23.871588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39f46d4-e23b-415e-909f-96f934a32e7b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:23.908400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:23.910527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:23.910527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:23.910527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:23.918246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:23.919698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:23.936046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:23.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:23.938795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 503/987] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:23.946868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:24.003683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:24.055536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938bd09e-a2e1-4023-b288-aa573d3361e1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:24.089277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:24.089936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:24.090220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:24.090495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:24.091277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:24.091583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:24.093024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6431f636-ffc3-4446-bba6-6fb66027a27b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:24.105474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 485/988] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:24.111886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:24.196602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:24.215529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e07b0bc-44ff-430a-ba35-760e523b36b7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:24.249690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:24.250886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:24.251165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:24.251436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:24.252260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] RBAC: Adding query filter params (role.id=4a90181c3f7847a7b2f8472e19bdd196, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2781978834a04fd4a48b4fb2ae2892d7) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:24.252565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:24.254023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8463697-3d34-4e33-bcc8-1f7ed41c133d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:24.269017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 504/989] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:24 2022] GET /identity/v3/role_assignments?user.id=2781978834a04fd4a48b4fb2ae2892d7&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=4a90181c3f7847a7b2f8472e19bdd196 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:25.485095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e06d4-1e45-4f85-acd7-1cf77b28cfd1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:25.485614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e06d4-1e45-4f85-acd7-1cf77b28cfd1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:25.485719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e06d4-1e45-4f85-acd7-1cf77b28cfd1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:25.487165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 486/990] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:25.492322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:25.492729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:25.492932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:25.510090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:25.511281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:25.580194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 505/991] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:25 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:25.602716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9e06d4-1e45-4f85-acd7-1cf77b28cfd1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:25.720968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9e06d4-1e45-4f85-acd7-1cf77b28cfd1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:25.730495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:25.757799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:25.758514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:25.758791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:25.759058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:25.759701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:25.761072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-85f25376-b37a-4d65-b9ec-ec8b9ee34740 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:25.772734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:25.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:25.779910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 487/992] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:25 2022] GET /identity/v3/services/compute => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:25.782000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:25.834428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8edecdb-19a6-46d0-bcb8-690387203e67 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:25.834711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8edecdb-19a6-46d0-bcb8-690387203e67 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:25.834987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8edecdb-19a6-46d0-bcb8-690387203e67 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:25.836485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 488/993] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:25.841159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:25.841456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:25.841735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:25.858357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:25.860395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:25.866376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:25.891873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c88e7-e113-4da6-9e5c-2cd814ed560f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:25.928898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:25.929560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:25.929845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:25.930115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:25.933697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:25.933697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:25.933697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:25.938998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 506/994] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:25 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:25.944432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:25.950056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 489/995] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:25 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:25.973071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.050195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e76550-813e-4d4f-8e60-75d23d574f31 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.080402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2afa392-7c1c-450b-ae2c-b4551a8d5db5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.082084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:26.086149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.086853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.087109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.087315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.088029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] RBAC: Adding query filter params (type=compute) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:26.088330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.089738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.100033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 507/996] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:25 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:26.101095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.105533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:26.123784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.124581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.124898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.125232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.125890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.127250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:26.139309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:26.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:26.141206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 490/997] 10.222.0.33 () {56 vars in 1126 bytes} [Tue Jul 19 10:52:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:26.146427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.211340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8d48df-afa0-403b-b05a-5afd43f993fd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.250633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f06b6-1876-4191-8dab-6ede2868d235 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.252268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.253118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.253420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.253703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.254496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] RBAC: Adding query filter params (interface=public, service_id=aff7178c47764d1e9090717e20287e4e, region_id=RegionOne) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:26.254809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.256172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 508/998] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/endpoints?service_id=aff7178c47764d1e9090717e20287e4e&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:26.267182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.289619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.290279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.290564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.290835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.291540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:26.291847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.293234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.298777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 491/999] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:26.304614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.336551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:26.337776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:26.373593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8a00f4-24aa-4eae-bbb9-aaeeb4eeba5a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.405700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7022b64-294e-401d-9234-501d195c7ac8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.407495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e2b2ab-19be-45e5-b24e-7e79ad731dbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.408146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e2b2ab-19be-45e5-b24e-7e79ad731dbc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.408424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e2b2ab-19be-45e5-b24e-7e79ad731dbc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.408720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e2b2ab-19be-45e5-b24e-7e79ad731dbc None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.409416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e2b2ab-19be-45e5-b24e-7e79ad731dbc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.410760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e2b2ab-19be-45e5-b24e-7e79ad731dbc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.417040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 509/1000] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 10:52:26.418056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a143e7-ebac-48b4-a78e-f50af98ef63f None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.418343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a143e7-ebac-48b4-a78e-f50af98ef63f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.418616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a143e7-ebac-48b4-a78e-f50af98ef63f None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.420208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 510/1001] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:26.426052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.426406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.426607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.443485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.444170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.444463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.444758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.445482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:26.445791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.447195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.447780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:26.449294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:26.454262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 492/1002] 10.222.0.33 () {56 vars in 1122 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:26.460076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.526977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 511/1003] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:26 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:26.550721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.582928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:26.591935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe16fdd-1631-419e-acd1-dc32edd8855a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.618448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:26.629807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.630470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.630755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.631031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.631774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.633168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.648914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 493/1004] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:26 2022] POST /identity/v3/endpoints => generated 325 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:52:26.664753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48356ffc-2b16-4fb8-accb-9dab0f0ebac7 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.707342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.708452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.708969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.709434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] PATH_INFO: `/v3/services/image` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.710267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.711842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:26.725113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:26.726175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:26.727019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:26.733743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 512/1005] 10.222.0.33 () {56 vars in 1105 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/services/image => generated 86 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:26.735914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:26.845195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eebbfa6-38f0-4174-96f5-2c633ad1f4e9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:26.879288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:26.879952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:26.880236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:26.880506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:26.881237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] RBAC: Adding query filter params (name=image) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:26.881541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:26.882890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3740cfc2-5355-4fc2-b508-5f1f963e9e50 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:26.892811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 494/1006] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:26.894334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:27.004060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e29e2e-483c-482a-a045-3b83204857b1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:27.026487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:27.038897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:27.039553 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:27.039831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:27.040099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:27.040828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] RBAC: Adding query filter params (type=image) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:27.041133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:27.042478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff51e6c3-a8f8-40f2-bcac-8a25c73515e7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:27.047226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 513/1007] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:26 2022] GET /identity/v3/services?type=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:28.461144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b3c2b-01f7-4d6c-9cbc-72e930ef33c8 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.461656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b3c2b-01f7-4d6c-9cbc-72e930ef33c8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.461780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b3c2b-01f7-4d6c-9cbc-72e930ef33c8 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.463245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 495/1008] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:28.468133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c507663-3710-4896-87d7-3bc4fbcf0638 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.468511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c507663-3710-4896-87d7-3bc4fbcf0638 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.468718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c507663-3710-4896-87d7-3bc4fbcf0638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.485109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8c507663-3710-4896-87d7-3bc4fbcf0638 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:28.486296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8c507663-3710-4896-87d7-3bc4fbcf0638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:28.552328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 514/1009] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:28 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:28.574910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b3c2b-01f7-4d6c-9cbc-72e930ef33c8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:28.594827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:28.681171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b3c2b-01f7-4d6c-9cbc-72e930ef33c8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:28.714587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:28.715292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.715573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.715844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.716489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:28.717890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:28.729421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bb8e0d-cf67-4539-891f-acbd96447e29 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.729716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bb8e0d-cf67-4539-891f-acbd96447e29 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.729996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bb8e0d-cf67-4539-891f-acbd96447e29 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:28.730214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:28.731170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 19 10:52:28.732088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:28.732862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 515/1010] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:28.736343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 496/1011] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:28 2022] GET /identity/v3/services/compute => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:28.736643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.736968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.737078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:28.737238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.754602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:28.755783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:28.826105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 516/1012] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:28 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:28.826704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:28.847232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675fb0d-8ddf-4181-91b3-dbb8ea431398 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:28.849877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:28.882103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:28.882765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.883045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.883318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.884441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] RBAC: Adding query filter params (name=compute) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:28.884441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:28.885702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:28.891344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 497/1013] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:28 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:28.896913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:28.897277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:28.950738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bebb30-8669-4c4b-bdfa-9beca1a17eec None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:28.987789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:28.988853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:28.989298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:28.989577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:28.990760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:28.992200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:28.996863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 517/1014] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:28.999583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:29.020766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7344da3a-8270-4627-b677-5e08e3f3f83d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:29.056121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:29.056831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:29.057106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:29.057379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:29.058085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] RBAC: Adding query filter params (type=compute) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:29.058388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:29.059748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:29.065854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 498/1015] 10.222.0.33 () {56 vars in 1112 bytes} [Tue Jul 19 10:52:28 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:29.071514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:29.085093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:29.114734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4ccf1-0c24-4450-be4f-a567caadd2b6 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:29.129171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:29.150147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5433f69d-efaf-43f0-a17f-427ce247c109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:29.150807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433f69d-efaf-43f0-a17f-427ce247c109 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:29.151092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433f69d-efaf-43f0-a17f-427ce247c109 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:29.151371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433f69d-efaf-43f0-a17f-427ce247c109 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:29.152074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5433f69d-efaf-43f0-a17f-427ce247c109 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:29.153539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5433f69d-efaf-43f0-a17f-427ce247c109 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:29.173391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f77bc-c902-4e33-b2db-030ab7b06a1e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:29.178972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 518/1016] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:28 2022] POST /identity/v3/projects => generated 311 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:29.180577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c41c16d-7e69-4da8-a996-a6c1f82e4e39 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:29.180906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c41c16d-7e69-4da8-a996-a6c1f82e4e39 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:29.181188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c41c16d-7e69-4da8-a996-a6c1f82e4e39 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:29.182592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 519/1017] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:29.189018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d6192-ecaa-4df1-b715-1606a2afbea5 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:29.189314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d6192-ecaa-4df1-b715-1606a2afbea5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:29.189599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d6192-ecaa-4df1-b715-1606a2afbea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:29.209556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-700d6192-ecaa-4df1-b715-1606a2afbea5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:29.210798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-700d6192-ecaa-4df1-b715-1606a2afbea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:29.215552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:29.216481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:29.216950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:29.217379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:29.218318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:29.220508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:29.243871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 499/1018] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:29 2022] POST /identity/v3/endpoints => generated 334 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:29.309303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 520/1019] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:29 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:29.333963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:29.408291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:29.446084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e4f061-0022-44bb-a3a9-5ecada822fb4 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:29.481585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ccb74e-c23f-4af1-a972-5599bfecf53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:29.482345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ccb74e-c23f-4af1-a972-5599bfecf53f None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:29.482759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ccb74e-c23f-4af1-a972-5599bfecf53f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:29.482970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ccb74e-c23f-4af1-a972-5599bfecf53f None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:29.483716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ccb74e-c23f-4af1-a972-5599bfecf53f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:29.485631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ccb74e-c23f-4af1-a972-5599bfecf53f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:29.501588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 500/1020] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:29 2022] POST /identity/v3/services => generated 242 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:31.296591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa7e8b-efda-4aa8-a9a5-3e20b213d184 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.297121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa7e8b-efda-4aa8-a9a5-3e20b213d184 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.297121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa7e8b-efda-4aa8-a9a5-3e20b213d184 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.298625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 521/1021] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:31.303724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2427ae-c3e8-40a8-9baf-a9bc465e83d2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.304121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2427ae-c3e8-40a8-9baf-a9bc465e83d2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.304291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2427ae-c3e8-40a8-9baf-a9bc465e83d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.321611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5f2427ae-c3e8-40a8-9baf-a9bc465e83d2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:31.322788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5f2427ae-c3e8-40a8-9baf-a9bc465e83d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:31.400768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 501/1022] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:31 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:31.419240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aa7e8b-efda-4aa8-a9a5-3e20b213d184 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:31.458976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51fa471-1ce5-4f87-8ea3-6235b4d0a3c4 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.459275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51fa471-1ce5-4f87-8ea3-6235b4d0a3c4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.459536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51fa471-1ce5-4f87-8ea3-6235b4d0a3c4 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.461077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 502/1023] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:31.465733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b63556-8338-48fe-b567-64bdb5176ea5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.466022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b63556-8338-48fe-b567-64bdb5176ea5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.466301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b63556-8338-48fe-b567-64bdb5176ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.483058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-41b63556-8338-48fe-b567-64bdb5176ea5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:31.484239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-41b63556-8338-48fe-b567-64bdb5176ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:31.517849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:31.530575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aa7e8b-efda-4aa8-a9a5-3e20b213d184 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:31.535872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:31.561113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 503/1024] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:31 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:31.569210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e24f5f-ee58-4d11-9810-ca9215d593a6 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.569559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e24f5f-ee58-4d11-9810-ca9215d593a6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.569771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e24f5f-ee58-4d11-9810-ca9215d593a6 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.571302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 504/1025] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:31.572129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:31.572941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.573222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.573491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.574646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:31.576097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.576201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:31.576381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.576652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.578017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 522/1026] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:31.581632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:31.594650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:31.595824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:31.667256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 505/1027] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:31 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:31.668228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:31.684102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1447ed2d-9206-40b1-a8e2-a58a52a43ffd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:31.719273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:31.720198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.720554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.720863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.730927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:31.742802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:31.743923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 523/1028] 10.222.0.33 () {56 vars in 1120 bytes} [Tue Jul 19 10:52:31 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:31.750563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:31.767015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:31.772549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9398a08a-f4a0-4f51-9ecd-0f67e9f2e677 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:31.786167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:31.808266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:31.808955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.809238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.809509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.810205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:31.811674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:31.831365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 506/1029] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:52:31 2022] POST /identity/v3/users => generated 285 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:31.832162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:31.845627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9721d53-6fc1-4828-9bfd-ecda666006a7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:31.888052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:31.888901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:31.889188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:31.889469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] PATH_INFO: `/v3/services/image` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:31.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:31.891584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:31.904131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:31.905288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:31.906191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 524/1030] 10.222.0.33 () {56 vars in 1105 bytes} [Tue Jul 19 10:52:31 2022] GET /identity/v3/services/image => generated 86 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:31.912239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:31.971064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b72f7e-c03e-4337-a61e-0c32271f5be9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.012667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.013378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.013632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.013738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.013931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.014683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:32.015079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.016446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.025003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 507/1031] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:52:31 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:32.030878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.033519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed31be6-f9e2-48dc-b5b1-43921630a657 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.068285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.076212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.077077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.077328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.077598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.078330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] RBAC: Adding query filter params (name=image) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:32.078632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.080087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.089450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 525/1032] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:31 2022] GET /identity/v3/services?name=image => generated 122 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:32.091184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.157100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30821e5b-1fe0-4529-8314-0611ddeced33 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.192121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.192322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2a3585-9e57-40b2-92cb-96166fb76595 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.192935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.193221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.193492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.204422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.205891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:32.223414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:32.224339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:32.225263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:32.225263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:32.225263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:32.225263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:32.225263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 508/1033] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:32.230267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.230392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.231046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.231298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.231571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.232274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] RBAC: Adding query filter params (type=image) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:32.232580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.235625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.241808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 526/1034] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/services?type=image => generated 350 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:32.247507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.262449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.318623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.339162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c7ce60-ffa0-46cd-922d-dd2f99a620cb None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.358525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fcca32-3d38-4e7a-800d-a0e01e2711cb None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.373996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.374654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.374966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.375240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.375960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:32.376267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.377643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.390053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 509/1035] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:32.396053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.400254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.400402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.400590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.400872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.401656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] RBAC: Adding query filter params (interface=public, service_id=fa5bd85d04cf47f6a98e342b071a553f, region_id=RegionOne) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:32.401967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.403326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.409882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 527/1036] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/endpoints?service_id=fa5bd85d04cf47f6a98e342b071a553f&interface=public®ion_id=RegionOne => generated 194 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:32.415441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.494940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.509633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afedcc23-ee1e-458f-bcd0-22ea1231d8ae None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.518050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a9b79c-2014-4aaa-9ccd-980384904e78 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.547165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.547950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.548231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.548502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.553829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.554480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.554760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.555032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.555706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.555937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.557093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.557384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.563947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 528/1037] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/services => generated 1957 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:32.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:32.573742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:32.574704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 510/1038] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:32.580643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.692820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b305313-df97-4d43-b85a-4a848ae17382 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.729814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.730482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.730760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.731028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.731846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:32.732144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.733578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.744027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.753263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 529/1039] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:32.758878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:32.761760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:32.860222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9681839-4872-4eec-916f-73c6648b478e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:32.894623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:32.895271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:32.895564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:32.895826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:32.971539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:32.973022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:32.975006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:33.089599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 511/1040] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:32 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:33.094774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:33.200145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a92b39-c4b8-4f0e-b41c-a75d3f562311 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:33.235272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:33.236047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.236330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.246864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:33.248300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:33.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:33.266939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:33.272847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:33.272847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:33.272847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:33.272847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:33.272847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 530/1041] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:33.273145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:33.371741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193fdd7-fd79-4054-93e8-f88986363680 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:33.373302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:33.410037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:33.410715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.410999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.411274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.412014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:33.412321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:33.413728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:33.430564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 512/1042] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:33.432188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:33.482549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:33.533370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed916f8-87aa-4cb6-805a-b8211b15de55 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:33.567429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69212597-7112-40ef-8853-eb718824339a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:33.568214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69212597-7112-40ef-8853-eb718824339a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.568497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69212597-7112-40ef-8853-eb718824339a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.568792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69212597-7112-40ef-8853-eb718824339a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.575390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69212597-7112-40ef-8853-eb718824339a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:33.576865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69212597-7112-40ef-8853-eb718824339a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-69212597-7112-40ef-8853-eb718824339a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:33.590455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:33.591340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:33.597473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 531/1043] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:33.599854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:33.704061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc00e8f-5f87-4b11-93d6-eaf9b395e94b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:33.738488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:33.739073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.739357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.739632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.740415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:33.740751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:33.742168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235e7357-416b-4421-8896-ac0546c7b305 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:33.758983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 513/1044] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:33.760310 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69212597-7112-40ef-8853-eb718824339a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:33.860684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69212597-7112-40ef-8853-eb718824339a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:33.871101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:33.896279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:33.897210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.897498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.897774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.898595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] RBAC: Adding query filter params (role.id=4a90181c3f7847a7b2f8472e19bdd196, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2a1c13591bab47d7ba589e14b83f969e) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:33.898909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:33.900327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e3a9b1-ebec-4fd0-8f7d-fab4ab260cdd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:33.912940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 532/1045] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:33 2022] GET /identity/v3/role_assignments?user.id=2a1c13591bab47d7ba589e14b83f969e&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=4a90181c3f7847a7b2f8472e19bdd196 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:33.932659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f3a10e-c312-48da-a1f5-e4a31d69269e None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.932996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f3a10e-c312-48da-a1f5-e4a31d69269e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.933251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f3a10e-c312-48da-a1f5-e4a31d69269e None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.934896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 514/1046] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:33.940504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:33.940861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:33.941120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:33.960243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:33.961492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:34.036327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 533/1047] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:33 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:34.059090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f3a10e-c312-48da-a1f5-e4a31d69269e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.085709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:34.166289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f3a10e-c312-48da-a1f5-e4a31d69269e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:34.199915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3485abea-e764-4a57-81b7-33341602069f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:34.200726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3485abea-e764-4a57-81b7-33341602069f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.201008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3485abea-e764-4a57-81b7-33341602069f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.201280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3485abea-e764-4a57-81b7-33341602069f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.212215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3485abea-e764-4a57-81b7-33341602069f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-3485abea-e764-4a57-81b7-33341602069f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:34.220498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:34.221455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:34.227397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 515/1048] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:34.229693 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.331122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf33a189-e3de-4944-a051-593de11bfe28 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:34.333372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:34.367791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:34.368463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.368771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.369201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.369955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:34.370259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:34.371615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:34.385239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 534/1049] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:34.386431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3485abea-e764-4a57-81b7-33341602069f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.472923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:34.492005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3485abea-e764-4a57-81b7-33341602069f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:34.526564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d2feae-e40d-46de-a25b-2e4064f8e470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:34.527426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d2feae-e40d-46de-a25b-2e4064f8e470 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.527708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d2feae-e40d-46de-a25b-2e4064f8e470 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.527981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d2feae-e40d-46de-a25b-2e4064f8e470 None admin] PATH_INFO: `/v3/users/7e7f4476c7c246cc9324e16cabb5f4a2` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.541665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d2feae-e40d-46de-a25b-2e4064f8e470 None admin] RBAC: Authorizing `identity:get_user(user_id=7e7f4476c7c246cc9324e16cabb5f4a2)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:34.543087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d2feae-e40d-46de-a25b-2e4064f8e470 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:34.544940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 516/1050] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/users/7e7f4476c7c246cc9324e16cabb5f4a2 => generated 285 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:34.550487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.599174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f0dc8-4e99-4391-b5f9-98f1b1922c67 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.599354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f0dc8-4e99-4391-b5f9-98f1b1922c67 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.599634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f0dc8-4e99-4391-b5f9-98f1b1922c67 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.601159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 517/1051] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:34.605835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.606123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.606396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.623659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:34.624880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:34.657509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390bb5d0-71e3-4990-b3cd-08c30a464127 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:34.691042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:34.691674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.691955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.692224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.693917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 518/1052] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:34 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:34.717742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.727598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:34.770088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:34.771534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:34.793796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:34.826040 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842f8b82-f504-48a4-9a89-69d999be0384 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:34.865329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:34.866038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:34.866316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:34.866771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] PATH_INFO: `/v3/services/image` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:34.867226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:34.868576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:34.876164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 535/1053] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:34.880299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:34.881531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 19 10:52:34.882383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:34.883139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.885588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 519/1054] 10.222.0.33 () {56 vars in 1105 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:34.887238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:34.955334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:34.997891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcdb925-93ce-4751-93df-2d8bcf5108b6 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:34.998998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c2bc92-131c-48e5-81a7-c11382b209ee None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:35.037930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:35.038787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.039197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.039657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.040063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:35.040756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.041042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.041202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] RBAC: Authorizing `identity:get_project(project_id=fa5963a416774bbdbe2a21614c6d1476)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:35.041579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.042288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] RBAC: Adding query filter params (name=image) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:35.042593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:35.042686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:35.043927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:35.049512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 536/1055] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476 => generated 311 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:35.049651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:35.050346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:35.054436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 520/1056] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:34 2022] GET /identity/v3/services?name=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:35.059888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:35.165261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bae988-5a4e-4fc0-b943-26c0eb383274 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:35.171994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ee7964-fc08-4527-a1e2-356ba8da9eee None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:35.200024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:35.200744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.201031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.201301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.202117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] RBAC: Adding query filter params (type=image) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:35.202529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:35.204482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:35.207537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:35.208177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.208458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.208823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.213951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 521/1057] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:35 2022] GET /identity/v3/services?type=image => generated 350 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:35.214088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=fa5963a416774bbdbe2a21614c6d1476, user.id=7e7f4476c7c246cc9324e16cabb5f4a2) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:35.214088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:35.214088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eb5613-fce7-4a4a-b775-ff4a53b272ae None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:35.214088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:35.214898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:35.228733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 537/1058] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:35 2022] GET /identity/v3/role_assignments?user.id=7e7f4476c7c246cc9324e16cabb5f4a2&scope.project.id=fa5963a416774bbdbe2a21614c6d1476&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:35.288170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:35.332131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff02ef-c07f-4e98-85da-cb54a1ace885 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:35.372334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0951c5-3ed5-46d5-83bf-1916db8ef446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:35.373021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0951c5-3ed5-46d5-83bf-1916db8ef446 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.373302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0951c5-3ed5-46d5-83bf-1916db8ef446 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.373576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0951c5-3ed5-46d5-83bf-1916db8ef446 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.374318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0951c5-3ed5-46d5-83bf-1916db8ef446 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:35.376514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0951c5-3ed5-46d5-83bf-1916db8ef446 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:35.396830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 522/1059] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:52:35 2022] POST /identity/v3/endpoints => generated 327 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:35.957265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a25279c-485c-40d8-9859-c09c12e9ed54 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.957748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a25279c-485c-40d8-9859-c09c12e9ed54 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.957860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a25279c-485c-40d8-9859-c09c12e9ed54 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.959236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 538/1060] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:35.964302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:35.964597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:35.964907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:35.981678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:35.982868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:36.058152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 523/1061] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:36.081462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a25279c-485c-40d8-9859-c09c12e9ed54 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:36.183229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a25279c-485c-40d8-9859-c09c12e9ed54 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:36.217386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:36.218119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:36.218403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:36.218703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:36.229247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:36.237423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:36.238624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:36.239468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:36.240751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 539/1062] 10.222.0.33 () {56 vars in 1120 bytes} [Tue Jul 19 10:52:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:36.245064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:36.291130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:36.349793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93facd90-76c6-46ee-b344-8a85aaa8fa34 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:36.383515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:36.384172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:36.384454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:36.384751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:36.385501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:36.385806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:36.387172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:36.416498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 524/1063] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:52:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:36.417656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:36.476180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:36.520641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f58e80-26da-4976-ad38-3cd159c9e1b9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:36.553781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:36.554557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:36.554837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:36.555104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:36.565963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:36.567387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:36.584510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:36.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:36.592898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:36.592898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:36.592898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:36.592898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:36.592898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 540/1064] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:36.593262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:36.696816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75e7b54-41ea-47c2-89d6-dc3d8a8f2480 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:36.705661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:36.733156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:36.733740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:36.734031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:36.734333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:36.735061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:36.735370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:36.736765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:36.749270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 525/1065] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:36.755236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:36.860108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2534a-869c-4760-9db9-aad5af6b6a9b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:36.870793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:36.895210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:36.895988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:36.896299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:36.896579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:36.904055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:36.905517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:36.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:36.920321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:36.926667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 541/1066] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:52:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:36.927260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.030934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78539e0-3ba1-4313-ac76-8e7c57db0d04 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.066997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.067667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.067948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.068221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.069097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:37.069405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.070810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d1de8e-4fa8-4751-ad21-2125e7fb3a30 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.088932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 526/1067] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:52:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:37.094898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.194870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e532d54-fb27-45be-8c5c-3a086ae2443e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.228227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9dc94-b950-4057-826a-fbf5671baae6 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.228537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9dc94-b950-4057-826a-fbf5671baae6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.228807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9dc94-b950-4057-826a-fbf5671baae6 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.229184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.229965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.230242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.230509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.231347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 527/1068] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:37.235277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.235565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.235838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.241829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.243333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.252957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:37.254145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:37.261801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:37.262674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:37.263568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:37.263568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:37.263568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:37.263568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:37.263568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 542/1069] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:37.269095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.275506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.294828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.330501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 528/1070] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:37 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:37.352094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.379929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62380a-5b5a-4776-bc3f-897a29d54f75 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.416611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.417402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.417690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.417965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.418689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:37.418994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.420359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.432417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 543/1071] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:37.438486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.454828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001e52c9-b326-44f2-b40c-6d4021be9815 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.488238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266303d1-9aaa-43f8-a83d-46602e794e0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.489041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266303d1-9aaa-43f8-a83d-46602e794e0b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.489329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266303d1-9aaa-43f8-a83d-46602e794e0b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.489600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266303d1-9aaa-43f8-a83d-46602e794e0b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.499872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266303d1-9aaa-43f8-a83d-46602e794e0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-266303d1-9aaa-43f8-a83d-46602e794e0b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:37.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:37.509164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:37.510054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 529/1072] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:37.511155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841c9065-42df-4003-8922-cf9454f5d4c1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.511155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841c9065-42df-4003-8922-cf9454f5d4c1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.511155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841c9065-42df-4003-8922-cf9454f5d4c1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.512444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 530/1073] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:37.515120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841c9065-42df-4003-8922-cf9454f5d4c1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.521226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.531826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.540259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d597715d-863b-44e9-8d44-945f1dd088ba None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.575554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.576270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.576551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.576855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.583886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.585359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-78d2b72b-ad5d-42ac-96cc-d1f49ba7bb1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:37.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:37.599670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 544/1074] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:37.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.601843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.601843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.618206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:37.619382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:37.623342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841c9065-42df-4003-8922-cf9454f5d4c1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.659164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.659816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.660099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.660372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.661149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:37.661451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.662790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.671052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 531/1075] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:37.672331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.690129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 545/1076] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:37 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:37.690951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.755608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.759502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.778152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133367c6-fdb5-4ff2-a7c4-c080fc883131 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.788731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711692d0-e211-42c5-a70b-ae42b3c9e394 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.811870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.813140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.813425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.813698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.814480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:37.814790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.816210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.822399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.823259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.823539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.823811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] PATH_INFO: `/v3/users/7e7f4476c7c246cc9324e16cabb5f4a2` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.825040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] RBAC: Authorizing `identity:get_user(user_id=7e7f4476c7c246cc9324e16cabb5f4a2)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.826530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.827595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 532/1077] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:37.828384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.832861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 546/1078] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/users/7e7f4476c7c246cc9324e16cabb5f4a2 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:37.833893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.928588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45d06e-8ad6-434d-bdde-38cf6c01f371 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.932671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61614e9e-87c6-43dd-b949-b7839a5722e1 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:37.964676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.965357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.965639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.965915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.966656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:37.967498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:37.967781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:37.968054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:37.969739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] RBAC: Authorizing `identity:get_project(project_id=fa5963a416774bbdbe2a21614c6d1476)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:37.969859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:37.971152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:37.972879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 547/1079] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:37.973617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:37.989248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:38.034848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:38.036300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:38.075562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcebd0e-f59b-41f2-95eb-5bb588b27ce3 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:38.110278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:38.111083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:38.111368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:38.111640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2a1c13591bab47d7ba589e14b83f969e/roles/4a90181c3f7847a7b2f8472e19bdd196` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:38.114115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=2a1c13591bab47d7ba589e14b83f969e, role_id=4a90181c3f7847a7b2f8472e19bdd196)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:38.115517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:38.129477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 548/1080] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:52:37 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/2a1c13591bab47d7ba589e14b83f969e/roles/4a90181c3f7847a7b2f8472e19bdd196 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:52:38.130287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:38.137354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 533/1081] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:37 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:38.143616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:38.186499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:38.211295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:38.253545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa6e3bc-969e-4a1d-b49f-017e81606d9d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:38.259619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1908a17-d9fd-49dd-8d36-c9b456667948 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:38.297622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:38.299126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:38.299126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:38.299126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476/users/7e7f4476c7c246cc9324e16cabb5f4a2/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:38.301230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] RBAC: Authorizing `identity:create_grant(project_id=fa5963a416774bbdbe2a21614c6d1476, user_id=7e7f4476c7c246cc9324e16cabb5f4a2, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:38.302667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:38.305304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:38.306539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:38.306829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:38.307102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:38.317057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 549/1082] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:52:38 2022] PUT /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476/users/7e7f4476c7c246cc9324e16cabb5f4a2/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:52:38.317996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:38.319452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1f97d256-d6f7-4863-9f75-f867a634edcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:38.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:38.339482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:38.345127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:38.345127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:38.345127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:38.345127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:38.345127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 534/1083] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:38.347184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:38.456618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf6f07-90f8-4e18-805f-5c1527506ce0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:38.488170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:38.497198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:38.498116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:38.498618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:38.498897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:38.499647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:38.499983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:38.501387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d0a08-4dba-4c39-80dd-806cd0f0db2a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:38.513487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 550/1084] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:40.188082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adebefd0-e1dd-4c71-bd0c-f905b6a92413 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.188584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adebefd0-e1dd-4c71-bd0c-f905b6a92413 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.188584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adebefd0-e1dd-4c71-bd0c-f905b6a92413 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.189951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 535/1085] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:40.195709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765cf698-dc01-41e5-b7bf-af6f89254610 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.196134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765cf698-dc01-41e5-b7bf-af6f89254610 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.196465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765cf698-dc01-41e5-b7bf-af6f89254610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.213419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-765cf698-dc01-41e5-b7bf-af6f89254610 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:40.214588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-765cf698-dc01-41e5-b7bf-af6f89254610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:40.280184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 551/1086] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:40.303269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adebefd0-e1dd-4c71-bd0c-f905b6a92413 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.355343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919dbf3-7f24-4c77-93ca-5812e0426965 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.355631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919dbf3-7f24-4c77-93ca-5812e0426965 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.355907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919dbf3-7f24-4c77-93ca-5812e0426965 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.357319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 552/1087] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:52:40.362108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14615e84-eede-4438-a958-1fa093e7790c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14615e84-eede-4438-a958-1fa093e7790c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.362671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14615e84-eede-4438-a958-1fa093e7790c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.379384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-14615e84-eede-4438-a958-1fa093e7790c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:40.380572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-14615e84-eede-4438-a958-1fa093e7790c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:40.391622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:40.405391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adebefd0-e1dd-4c71-bd0c-f905b6a92413 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.439942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.440673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.440988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.441274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.446452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 553/1088] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:52:40.451318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:40.460032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:40.461104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:40.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 536/1089] 10.222.0.33 () {56 vars in 1120 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:40.467064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14615e84-eede-4438-a958-1fa093e7790c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.470548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.514448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:40.564682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14615e84-eede-4438-a958-1fa093e7790c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.568929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe85ca18-4dca-4903-9f7b-5d65fcd3c32f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.597856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.598515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.598799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.599075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.599815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:40.600121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:40.601515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:40.602363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.603137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.603680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.603680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.614688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 554/1090] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:40.614827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:40.615974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:40.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:40.623137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:40.623975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 537/1091] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:40.629805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.640875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:40.715929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7d950-ff20-485b-a251-ea7b06d991d9 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.725821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:40.729908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aab470d-15ef-4961-9ef4-24ab418d3953 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.749777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.751093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.751370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.751638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.761866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:40.763309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:40.763767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.764420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.764718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.765008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.765745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:40.766048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:40.767384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:40.775654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 538/1092] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:40.780000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:40.780884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:40.781817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:40.781817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:40.781817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:40.781817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:40.782036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.782113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 555/1093] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:52:40.786854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.850952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:40.883905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519eb9fe-a42f-4f08-8ef4-44be757527fc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.884069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c784071-e068-45fa-a6b4-0b4226c68e2f None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:40.916194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.916498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:40.916882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.917172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.917389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:40.917499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.917668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:40.917945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] PATH_INFO: `/v3/users/7e7f4476c7c246cc9324e16cabb5f4a2` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:40.918160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:40.918516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:40.919166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] RBAC: Authorizing `identity:get_user(user_id=7e7f4476c7c246cc9324e16cabb5f4a2)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:40.919837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:40.920631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:40.922824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 539/1094] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/users/7e7f4476c7c246cc9324e16cabb5f4a2 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:40.928374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.940434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 556/1095] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:40.941476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:40.964215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.028597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bf9819-d979-443b-a1f6-0ecfdc5cd45e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.041937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f75fc1e-6efa-4c36-a64b-457dff839360 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.063005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.063647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.063928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.064196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.073543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.075636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.076427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.076725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.077007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.084189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.085659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:52:41.102155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:41.103230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 19 10:52:41.104165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:41.104939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 557/1096] 10.222.0.33 () {56 vars in 1093 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:41.109091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.145183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.146642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.189772 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.207448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d839fd2e-5b27-4210-a5e6-73f81973b816 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.241396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.242047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.242327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.242604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.243422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:41.243729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.245164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.245943 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 540/1097] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:40 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 19 10:52:41.251652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.263133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 558/1098] 10.222.0.33 () {56 vars in 1133 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:41.268638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.290090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.350743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b75c1-e946-4f8d-bc50-7b79f507627e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.372367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf7118c-5738-46aa-9ab5-b7890d01c1cc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.405496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.406979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.407264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.407541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.408761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] RBAC: Authorizing `identity:get_project(project_id=fa5963a416774bbdbe2a21614c6d1476)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.410198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.411441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.411902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 541/1099] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476 => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:41.412157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.412439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.412741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.417907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.424002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.480290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.481786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.514788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576343d3-9bb6-428c-b210-c53554285633 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.522355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.549627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.550208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.550486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.550758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.551582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=fa5963a416774bbdbe2a21614c6d1476, user.id=7e7f4476c7c246cc9324e16cabb5f4a2) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:41.551892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.553331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.559059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 542/1100] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/role_assignments?user.id=7e7f4476c7c246cc9324e16cabb5f4a2&scope.project.id=fa5963a416774bbdbe2a21614c6d1476&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:52:41.580173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 559/1101] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 19 10:52:41.586141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.690188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7564e-42df-4854-a8bd-3654d4ec2212 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.724921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.725703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.725983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.726254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.736862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.738545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:52:41.755844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:41.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:41.758113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:52:41.758113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:52:41.758113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:52:41.758113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:41.758113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 543/1102] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:41.762823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.803167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:41.861719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce95c6f-c664-44be-bb24-992a78005614 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:41.895335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:41.895988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:41.896294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:41.896563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:41.897313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:41.897645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:41.899010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:41.914533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 560/1103] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:41.915731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:41.942683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:42.019070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca48139d-7617-4aff-843e-cd5c8698153d None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:42.051214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:42.051997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:42.052280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:42.052555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:42.059247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:42.060618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:52:42.074944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:42.075954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:52:42.076892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:52:42.080849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 544/1104] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:52:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:42.082466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:42.177268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:42.180329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9cfbf-a1a8-494b-b5e9-0f3269543ee8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:42.213554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:42.214204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:42.214482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:42.214752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:42.215527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:42.215862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:42.217342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4e6c81-0ebb-41d6-9985-6ab5c4bc8156 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:42.229180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 561/1105] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:52:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:42.235629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:42.315300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:42.333178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b99df7b-75f8-4343-a4c9-eed95eb90be3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:42.366366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:42.367008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:42.367294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:42.367566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:42.368389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] RBAC: Adding query filter params (role.id=4a90181c3f7847a7b2f8472e19bdd196, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=2a1c13591bab47d7ba589e14b83f969e) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:42.368721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:42.370140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8e7a03-8b82-48a7-b386-cce7c253f7a7 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:42.376558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 545/1106] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:42 2022] GET /identity/v3/role_assignments?user.id=2a1c13591bab47d7ba589e14b83f969e&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=4a90181c3f7847a7b2f8472e19bdd196 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:43.617086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e295c885-108c-476f-8362-9b9519e3aa1f None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:43.617609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e295c885-108c-476f-8362-9b9519e3aa1f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:43.617694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e295c885-108c-476f-8362-9b9519e3aa1f None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:43.619102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 562/1107] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:43.624236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:43.624630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:43.624809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:43.642139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:43.643313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:43.706185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 546/1108] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:43.729662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e295c885-108c-476f-8362-9b9519e3aa1f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:43.827730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e295c885-108c-476f-8362-9b9519e3aa1f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:43.859943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69c588c-362e-4ab7-b571-fabfb8a29445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:43.860722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69c588c-362e-4ab7-b571-fabfb8a29445 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:43.861009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69c588c-362e-4ab7-b571-fabfb8a29445 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:43.861280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69c588c-362e-4ab7-b571-fabfb8a29445 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:43.862433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69c588c-362e-4ab7-b571-fabfb8a29445 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:43.864134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69c588c-362e-4ab7-b571-fabfb8a29445 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:43.865799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 563/1109] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:43.871170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:43.946141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:43.968341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad82c7c2-4218-441e-a0ad-2bd07774b28f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:44.001616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e366764-1c97-4dec-8775-288efd5066b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:44.002267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e366764-1c97-4dec-8775-288efd5066b3 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:44.002546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e366764-1c97-4dec-8775-288efd5066b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:44.002816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e366764-1c97-4dec-8775-288efd5066b3 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:44.003509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e366764-1c97-4dec-8775-288efd5066b3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:44.004938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e366764-1c97-4dec-8775-288efd5066b3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:44.023042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 547/1110] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:52:43 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:46.044135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6311268d-30eb-4885-86ab-72fe98693eae None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:46.044649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6311268d-30eb-4885-86ab-72fe98693eae None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:46.044793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6311268d-30eb-4885-86ab-72fe98693eae None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:46.046193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 564/1111] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:46.051756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:46.052128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:46.052311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:46.069189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:46.070367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:46.134513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 548/1112] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:46 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:46.157281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6311268d-30eb-4885-86ab-72fe98693eae None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:46.184933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:46.256484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6311268d-30eb-4885-86ab-72fe98693eae None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:46.289141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:46.289973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:46.290252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:46.290521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] PATH_INFO: `/v3/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:46.291956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] RBAC: Authorizing `identity:get_role(role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:46.293329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:46.294652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 565/1113] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:46 2022] GET /identity/v3/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:46.300918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:46.358125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:46.396481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ca092a-7460-43f7-88db-670fe5154717 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:46.431867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:46.432736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:46.433019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:46.433288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] PATH_INFO: `/v3/users/30439b91a16645a0a25133c829108c1b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:46.446507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] RBAC: Authorizing `identity:get_user(user_id=30439b91a16645a0a25133c829108c1b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:46.447864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:46.449511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 549/1114] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:46 2022] GET /identity/v3/users/30439b91a16645a0a25133c829108c1b => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:46.455827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:46.546325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:46.552093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece70246-3b5d-4c80-aa35-40a43e7cebd7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:46.585285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:46.585912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:46.586198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:46.586476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:46.651134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:46.652590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:46.740956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:46.747471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 566/1115] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:46 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:46.753837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:46.849409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7463aed9-5da4-4f61-aa07-28158e15f317 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:46.869227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:46.883438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c536570-dabb-4f63-b364-e331f27cda52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:46.884283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c536570-dabb-4f63-b364-e331f27cda52 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:46.884563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c536570-dabb-4f63-b364-e331f27cda52 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:46.884865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c536570-dabb-4f63-b364-e331f27cda52 None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:46.886031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c536570-dabb-4f63-b364-e331f27cda52 None admin] RBAC: Authorizing `identity:get_project(project_id=fa5963a416774bbdbe2a21614c6d1476)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:46.887442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c536570-dabb-4f63-b364-e331f27cda52 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:46.889153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 550/1116] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:52:46 2022] GET /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:46.895261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:46.991447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37483a19-2bdd-4a36-9ef8-ea5bcc5db39f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:47.023566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:47.024204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:47.024490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:47.024794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:47.025621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] RBAC: Adding query filter params (role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=fa5963a416774bbdbe2a21614c6d1476, user.id=30439b91a16645a0a25133c829108c1b) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:47.025933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:47.027341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf185fa-4d02-4894-8e6f-b441dad3aee8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:47.033104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 567/1117] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:46 2022] GET /identity/v3/role_assignments?user.id=30439b91a16645a0a25133c829108c1b&scope.project.id=fa5963a416774bbdbe2a21614c6d1476&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:48.994804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0388942a-4e63-47ae-bc38-58f81bd06636 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:48.995681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0388942a-4e63-47ae-bc38-58f81bd06636 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:48.995963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0388942a-4e63-47ae-bc38-58f81bd06636 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:48.997418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 551/1118] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:49.002867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:49.003292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:49.003423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:49.020153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:49.021391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:49.078581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:49.090006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 568/1119] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:49 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:49.112778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0388942a-4e63-47ae-bc38-58f81bd06636 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:49.211937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0388942a-4e63-47ae-bc38-58f81bd06636 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:49.244764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:49.245601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:49.245883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:49.246156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] PATH_INFO: `/v3/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:49.247612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] RBAC: Authorizing `identity:get_role(role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:49.248988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:49.250853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 552/1120] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:49 2022] GET /identity/v3/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:49.256677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:49.351828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2be91-202b-4368-a251-9345b3e16681 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:49.385801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:49.386649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:49.386933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:49.387207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] PATH_INFO: `/v3/users/30439b91a16645a0a25133c829108c1b` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:49.388366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] RBAC: Authorizing `identity:get_user(user_id=30439b91a16645a0a25133c829108c1b)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:49.389821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:49.392030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 569/1121] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:49 2022] GET /identity/v3/users/30439b91a16645a0a25133c829108c1b => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:49.397618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:49.452986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:49.499044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0561f9-d424-4490-97cd-6089703c84b1 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:49.533143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a44a561-540d-40a7-b5e1-9ab0972805b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:49.533994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44a561-540d-40a7-b5e1-9ab0972805b4 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:49.534275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44a561-540d-40a7-b5e1-9ab0972805b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:49.534547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44a561-540d-40a7-b5e1-9ab0972805b4 None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:49.535710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a44a561-540d-40a7-b5e1-9ab0972805b4 None admin] RBAC: Authorizing `identity:get_project(project_id=fa5963a416774bbdbe2a21614c6d1476)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:49.537165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a44a561-540d-40a7-b5e1-9ab0972805b4 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:49.538881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 553/1122] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:52:49 2022] GET /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:49.545371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:49.602777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:49.646598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e69901-4d60-4cb2-bbb9-3ddfbdd6538c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:49.679872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49e6be4-9901-4a2d-9844-0fdc7db68abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:49.680914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49e6be4-9901-4a2d-9844-0fdc7db68abd None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:49.681189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49e6be4-9901-4a2d-9844-0fdc7db68abd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:49.681463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49e6be4-9901-4a2d-9844-0fdc7db68abd None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476/users/30439b91a16645a0a25133c829108c1b/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:49.683727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49e6be4-9901-4a2d-9844-0fdc7db68abd None admin] RBAC: Authorizing `identity:create_grant(project_id=fa5963a416774bbdbe2a21614c6d1476, user_id=30439b91a16645a0a25133c829108c1b, role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:49.685177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49e6be4-9901-4a2d-9844-0fdc7db68abd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:49.698235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 570/1123] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:52:49 2022] PUT /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476/users/30439b91a16645a0a25133c829108c1b/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 10:52:51.672522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6493b75-6937-4160-b48c-3b7b671a4c04 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:51.673079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6493b75-6937-4160-b48c-3b7b671a4c04 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:51.673079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6493b75-6937-4160-b48c-3b7b671a4c04 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:51.674461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 554/1124] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:51.679609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:51.680048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:51.680155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:51.696957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:51.698138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:51.762320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 571/1125] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:51 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:51.785654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6493b75-6937-4160-b48c-3b7b671a4c04 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:51.883912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6493b75-6937-4160-b48c-3b7b671a4c04 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:51.905905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:51.916846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:51.917690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:51.917970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:51.918239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] PATH_INFO: `/v3/roles/e9e791a4bf924be58d52b6ba8dbf07d0` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:51.919698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] RBAC: Authorizing `identity:get_role(role_id=e9e791a4bf924be58d52b6ba8dbf07d0)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:51.921094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:51.923012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 555/1126] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:51 2022] GET /identity/v3/roles/e9e791a4bf924be58d52b6ba8dbf07d0 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:51.928777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:51.976170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:52.026291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f61ebc3-30fb-4eb7-a6b4-47b2d0a3d03f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:52.059101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:52.059952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:52.060236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:52.060511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] PATH_INFO: `/v3/users/30439b91a16645a0a25133c829108c1b` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:52.061701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] RBAC: Authorizing `identity:get_user(user_id=30439b91a16645a0a25133c829108c1b)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:52.063886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:52.066285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 572/1127] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:51 2022] GET /identity/v3/users/30439b91a16645a0a25133c829108c1b => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:52.072096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:52.168733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f17986f-90e4-4373-8b8e-4c15756380bd None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:52.202176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:52.202746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:52.203059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:52.203335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:52.259581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:52.269083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:52.270511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:52.364951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 556/1128] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:52:52 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:52:52.371230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:52.465858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8958e137-917f-4d0a-a1ee-cc738b6f6af2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:52.481692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:52.499577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256bea40-3a9b-43b2-b392-ec020c965f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:52.500457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256bea40-3a9b-43b2-b392-ec020c965f09 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:52.500771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256bea40-3a9b-43b2-b392-ec020c965f09 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:52.501042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256bea40-3a9b-43b2-b392-ec020c965f09 None admin] PATH_INFO: `/v3/projects/fa5963a416774bbdbe2a21614c6d1476` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:52.502192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256bea40-3a9b-43b2-b392-ec020c965f09 None admin] RBAC: Authorizing `identity:get_project(project_id=fa5963a416774bbdbe2a21614c6d1476)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:52.503599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256bea40-3a9b-43b2-b392-ec020c965f09 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:52.505285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 573/1129] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:52:52 2022] GET /identity/v3/projects/fa5963a416774bbdbe2a21614c6d1476 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:52.511396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:52.598899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:52.607919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60c2845-22fb-4736-b5dd-fc21c7815ce5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:52.641773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:52.642406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:52.642690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:52.642966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:52.643798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] RBAC: Adding query filter params (role.id=e9e791a4bf924be58d52b6ba8dbf07d0, scope.project.id=fa5963a416774bbdbe2a21614c6d1476, user.id=30439b91a16645a0a25133c829108c1b) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:52.644112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:52.645550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aaf508-3e59-4aa7-8e61-fec294bd595d None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:52.651195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 557/1130] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:52:52 2022] GET /identity/v3/role_assignments?user.id=30439b91a16645a0a25133c829108c1b&scope.project.id=fa5963a416774bbdbe2a21614c6d1476&role.id=e9e791a4bf924be58d52b6ba8dbf07d0 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:54.652423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8a5e6a-2834-4a35-b3a9-d8cc1c9236cd None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:54.654287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8a5e6a-2834-4a35-b3a9-d8cc1c9236cd None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:54.654287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8a5e6a-2834-4a35-b3a9-d8cc1c9236cd None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:54.654472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 574/1131] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:54.659667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:54.660108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:54.660229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:54.676631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:54.677845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:54.742180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 558/1132] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:54.765387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8a5e6a-2834-4a35-b3a9-d8cc1c9236cd None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:54.862028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8a5e6a-2834-4a35-b3a9-d8cc1c9236cd None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:54.894319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4660d75-74bd-4036-b678-bd34f4717d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:54.895079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4660d75-74bd-4036-b678-bd34f4717d8b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:54.895358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4660d75-74bd-4036-b678-bd34f4717d8b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:54.895629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4660d75-74bd-4036-b678-bd34f4717d8b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:54.896793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4660d75-74bd-4036-b678-bd34f4717d8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:54.898199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4660d75-74bd-4036-b678-bd34f4717d8b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:54.899764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 575/1133] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:54.905456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:54.986894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:55.001960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51484e33-dfa6-44aa-9af0-743468d7285e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:55.034048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0b00e9-71f7-4b0a-a021-1ef9307c1030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:55.034691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0b00e9-71f7-4b0a-a021-1ef9307c1030 None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:55.034974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0b00e9-71f7-4b0a-a021-1ef9307c1030 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:55.035249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0b00e9-71f7-4b0a-a021-1ef9307c1030 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:55.035965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0b00e9-71f7-4b0a-a021-1ef9307c1030 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:55.037439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0b00e9-71f7-4b0a-a021-1ef9307c1030 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:55.063472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 559/1134] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:52:54 2022] POST /identity/v3/projects => generated 311 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:52:57.093681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4a44c-a572-4e79-9ee4-79ab918f2792 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:57.094216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4a44c-a572-4e79-9ee4-79ab918f2792 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:57.094216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4a44c-a572-4e79-9ee4-79ab918f2792 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:57.095576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 576/1135] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:57.100845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:57.101223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:57.101403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:57.118437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:57.119607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:57.183067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 560/1136] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:57.208251 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4a44c-a572-4e79-9ee4-79ab918f2792 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:57.260966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:57.306298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4a44c-a572-4e79-9ee4-79ab918f2792 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:57.338531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99450366-78b6-4099-917d-42c75b0ea103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:57.339298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99450366-78b6-4099-917d-42c75b0ea103 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:57.339583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99450366-78b6-4099-917d-42c75b0ea103 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:57.339856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99450366-78b6-4099-917d-42c75b0ea103 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:57.341037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99450366-78b6-4099-917d-42c75b0ea103 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:57.342458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99450366-78b6-4099-917d-42c75b0ea103 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:57.344079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 577/1137] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:52:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:57.349757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:57.412602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:57.447499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b8d47a-9620-474b-9452-f369400c25a8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:57.480714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178b609-064a-4899-9cc5-11be885783bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:57.481344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178b609-064a-4899-9cc5-11be885783bd None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:57.481626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178b609-064a-4899-9cc5-11be885783bd None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:57.481897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178b609-064a-4899-9cc5-11be885783bd None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:57.482606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178b609-064a-4899-9cc5-11be885783bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:57.484025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178b609-064a-4899-9cc5-11be885783bd None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:57.504139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 561/1138] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:52:57 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:52:59.526209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368edb41-ccf3-48ed-b790-410b17d0ed92 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:59.526753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368edb41-ccf3-48ed-b790-410b17d0ed92 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:59.526753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368edb41-ccf3-48ed-b790-410b17d0ed92 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:59.528150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 578/1139] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:52:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:52:59.534025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:59.534419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:59.534584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:59.550957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:52:59.552155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:52:59.617437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 562/1140] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:52:59 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:52:59.641140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368edb41-ccf3-48ed-b790-410b17d0ed92 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:59.740199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368edb41-ccf3-48ed-b790-410b17d0ed92 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:59.759732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:59.773815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:59.774584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:59.774870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:59.775145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:59.785053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:52:59.792605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:52:59.793773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:52:59.795022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 579/1141] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:52:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:52:59.799590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:52:59.852815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:52:59.897475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1775ba-8cff-4f6e-8432-d3d982b6c551 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:52:59.929759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:52:59.930412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:52:59.930691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:52:59.930961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:52:59.931704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:52:59.932005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:52:59.933739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:52:59.941791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 563/1142] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:52:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:52:59.947560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:00.042857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f770768c-0031-4161-8eb4-10d9b18c6109 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:00.074806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:00.075644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:00.075918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:00.076183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] PATH_INFO: `/v3/users/722b2aad5c3e4453a3c4de593ef9e46d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:00.089288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] RBAC: Authorizing `identity:get_user(user_id=722b2aad5c3e4453a3c4de593ef9e46d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:00.090703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:00.092298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 580/1143] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:52:59 2022] GET /identity/v3/users/722b2aad5c3e4453a3c4de593ef9e46d => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:00.097784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:00.194041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffac154-13c4-406c-9ab0-d249bb9f3676 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:00.209112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:00.226343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:00.226970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:00.227250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:00.227528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:00.291367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:00.292798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:00.386339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 564/1144] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:53:00 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:53:00.392126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:00.430910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:00.489080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f007391d-af70-4361-8c11-f1c45b4b6a4b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:00.520822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50df9ce-e9fa-468c-b2ed-e50b47f09f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:00.521655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50df9ce-e9fa-468c-b2ed-e50b47f09f67 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:00.521933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50df9ce-e9fa-468c-b2ed-e50b47f09f67 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:00.522205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50df9ce-e9fa-468c-b2ed-e50b47f09f67 None admin] PATH_INFO: `/v3/projects/b6b46caeee494311bbea050d2fdd06da` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:00.523359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50df9ce-e9fa-468c-b2ed-e50b47f09f67 None admin] RBAC: Authorizing `identity:get_project(project_id=b6b46caeee494311bbea050d2fdd06da)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:00.524790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50df9ce-e9fa-468c-b2ed-e50b47f09f67 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:00.526446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 581/1145] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:53:00 2022] GET /identity/v3/projects/b6b46caeee494311bbea050d2fdd06da => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:00.532952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:00.550170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:00.628100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ca4073-53fb-429d-b914-78a73241078b None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:00.660554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:00.661154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:00.661435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:00.661706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:00.662515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=b6b46caeee494311bbea050d2fdd06da, user.id=722b2aad5c3e4453a3c4de593ef9e46d) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:00.662823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:00.664211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f2b3ed-0004-4941-8399-9ae7c43f7d53 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:00.669448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 565/1146] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:53:00 2022] GET /identity/v3/role_assignments?user.id=722b2aad5c3e4453a3c4de593ef9e46d&scope.project.id=b6b46caeee494311bbea050d2fdd06da&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:02.656640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ca902-c914-49a2-89ed-42e28825a179 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:02.657185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ca902-c914-49a2-89ed-42e28825a179 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:02.657275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ca902-c914-49a2-89ed-42e28825a179 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:02.658600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 582/1147] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:02.663689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:02.664137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:02.664239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:02.681767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:02.682944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:02.747409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 566/1148] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:02 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:02.769333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29ca902-c914-49a2-89ed-42e28825a179 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:02.868893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29ca902-c914-49a2-89ed-42e28825a179 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:02.875870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:02.901584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:02.902353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:02.902634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:02.902906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:02.913108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:53:02.920592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:53:02.921595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:53:02.923205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 583/1149] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:53:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:53:02.928361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:02.940819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:03.028601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67468060-8fdc-443a-b602-4bfa536144d5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:03.062739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:03.063398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:03.063683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:03.063957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:03.065213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:03.065524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:03.066885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:03.074781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 567/1150] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:53:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:03.080806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:03.178034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9594cae-8362-41e8-8720-518849357d6f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:03.210328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:03.211183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:03.211464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:03.211737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] PATH_INFO: `/v3/users/722b2aad5c3e4453a3c4de593ef9e46d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:03.212927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] RBAC: Authorizing `identity:get_user(user_id=722b2aad5c3e4453a3c4de593ef9e46d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:03.214334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:03.215937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 584/1151] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:53:03 2022] GET /identity/v3/users/722b2aad5c3e4453a3c4de593ef9e46d => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:03.222475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:03.301891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:03.332405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d952905-89f0-477c-a9f0-7d6879622a15 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:03.367002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61d6c1a-40e1-41bc-af8a-e8d50f461be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:03.367847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d6c1a-40e1-41bc-af8a-e8d50f461be6 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:03.368129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d6c1a-40e1-41bc-af8a-e8d50f461be6 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:03.368400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d6c1a-40e1-41bc-af8a-e8d50f461be6 None admin] PATH_INFO: `/v3/projects/b6b46caeee494311bbea050d2fdd06da` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:03.369625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61d6c1a-40e1-41bc-af8a-e8d50f461be6 None admin] RBAC: Authorizing `identity:get_project(project_id=b6b46caeee494311bbea050d2fdd06da)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:03.371048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61d6c1a-40e1-41bc-af8a-e8d50f461be6 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:03.372772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 568/1152] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:53:03 2022] GET /identity/v3/projects/b6b46caeee494311bbea050d2fdd06da => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:03.378782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:03.413231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:03.477953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b77d2f-2101-4c0b-ad68-d2139e4bb752 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:03.510229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341094b6-1c58-4433-a9d8-a7e37d3f2865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:03.511094 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341094b6-1c58-4433-a9d8-a7e37d3f2865 None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:03.511371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341094b6-1c58-4433-a9d8-a7e37d3f2865 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:03.511640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341094b6-1c58-4433-a9d8-a7e37d3f2865 None admin] PATH_INFO: `/v3/projects/b6b46caeee494311bbea050d2fdd06da/users/722b2aad5c3e4453a3c4de593ef9e46d/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:03.513566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341094b6-1c58-4433-a9d8-a7e37d3f2865 None admin] RBAC: Authorizing `identity:create_grant(project_id=b6b46caeee494311bbea050d2fdd06da, user_id=722b2aad5c3e4453a3c4de593ef9e46d, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:03.515001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341094b6-1c58-4433-a9d8-a7e37d3f2865 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:03.528493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 585/1153] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:53:03 2022] PUT /identity/v3/projects/b6b46caeee494311bbea050d2fdd06da/users/722b2aad5c3e4453a3c4de593ef9e46d/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:53:05.527721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cf979c-ed4e-4009-9c63-e1d6c55c3c3e None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:05.528275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cf979c-ed4e-4009-9c63-e1d6c55c3c3e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:05.528275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cf979c-ed4e-4009-9c63-e1d6c55c3c3e None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:05.529672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 569/1154] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:05.535346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:05.535709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:05.535903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:05.552260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:05.553470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:05.617431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 586/1155] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:05 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:05.639964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cf979c-ed4e-4009-9c63-e1d6c55c3c3e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:05.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cf979c-ed4e-4009-9c63-e1d6c55c3c3e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:05.773591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:05.774359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:05.774648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:05.774924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:05.782327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:05.785891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:53:05.793561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:53:05.794709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:53:05.795732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 570/1156] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:53:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:53:05.800913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:05.825687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:05.898792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d6a18-46be-4613-8002-0de7baabacc3 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:05.930928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:05.931589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:05.931873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:05.932145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:05.932918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:05.933222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:05.934774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:05.943068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 587/1157] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:53:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:05.948905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:06.045332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459fdd82-64ef-4033-8427-389ebdb16971 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:06.077900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:06.079326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:06.079607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:06.079876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] PATH_INFO: `/v3/users/722b2aad5c3e4453a3c4de593ef9e46d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:06.081080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] RBAC: Authorizing `identity:get_user(user_id=722b2aad5c3e4453a3c4de593ef9e46d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:06.082487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:06.084067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 571/1158] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:53:05 2022] GET /identity/v3/users/722b2aad5c3e4453a3c4de593ef9e46d => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:06.089528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:06.173924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:06.189595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503590dc-f664-4bf8-918c-db91ae5a3432 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:06.221297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:06.221937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:06.222223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:06.222499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:06.287389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:06.288859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:06.368535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:06.384024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 588/1159] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:53:06 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:53:06.389850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:06.454586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:06.487447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0376cb6-1aef-4fc6-9d87-fe2891e87b2c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:06.520830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271cc3c5-b6dc-4007-aeb1-a21aef76a0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:06.521658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cc3c5-b6dc-4007-aeb1-a21aef76a0b9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:06.521934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cc3c5-b6dc-4007-aeb1-a21aef76a0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:06.522198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cc3c5-b6dc-4007-aeb1-a21aef76a0b9 None admin] PATH_INFO: `/v3/projects/b6b46caeee494311bbea050d2fdd06da` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:06.523350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271cc3c5-b6dc-4007-aeb1-a21aef76a0b9 None admin] RBAC: Authorizing `identity:get_project(project_id=b6b46caeee494311bbea050d2fdd06da)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:06.524781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271cc3c5-b6dc-4007-aeb1-a21aef76a0b9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:06.526453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 572/1160] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:53:06 2022] GET /identity/v3/projects/b6b46caeee494311bbea050d2fdd06da => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:06.533039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:06.626266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e309ef-4c72-41f9-b61d-7c312bb7a0bd None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:06.658435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:06.659052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:06.659334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:06.659607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:06.660418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=b6b46caeee494311bbea050d2fdd06da, user.id=722b2aad5c3e4453a3c4de593ef9e46d) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:06.660751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:06.662139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d28d8-2039-483b-826e-cd45fb7f2e25 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:06.667678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 589/1161] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:53:06 2022] GET /identity/v3/role_assignments?user.id=722b2aad5c3e4453a3c4de593ef9e46d&scope.project.id=b6b46caeee494311bbea050d2fdd06da&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:08.678348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a41e1-f7fb-489a-8610-0b3827956546 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:08.678904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a41e1-f7fb-489a-8610-0b3827956546 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:08.678904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a41e1-f7fb-489a-8610-0b3827956546 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:08.680250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 573/1162] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:08.685312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:08.685706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:08.685864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:08.702692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:08.704391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:08.748117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:08.769031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 590/1163] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:08 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:08.790913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167a41e1-f7fb-489a-8610-0b3827956546 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:08.890389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167a41e1-f7fb-489a-8610-0b3827956546 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:08.923748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:08.924380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:08.924667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:08.924971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:08.928322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:08.991457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:08.992922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:09.088704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 574/1164] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:53:08 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:53:09.094636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:09.189990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746b8177-c7de-4a66-8215-f6af2279bce8 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:09.221721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:09.222501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:09.222778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:09.223049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:09.229519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:09.230940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-3af99b75-5491-4862-a4e0-05c7b81e750a None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:53:09.250579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:53:09.251755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:53:09.252932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 591/1165] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:53:09 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:53:09.258444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:09.291633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:09.354708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe86937-ff97-4d76-bc4a-cb22bb0c8718 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:09.388576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:09.389248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:09.389535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:09.389811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:09.390529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:09.390836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:09.392184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a63d304-60e3-49e6-9471-cab38ae38970 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:09.399192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 575/1166] 10.222.0.33 () {56 vars in 1115 bytes} [Tue Jul 19 10:53:09 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:11.389821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed5d940-1ed9-4cb6-8b1d-66a2aeed33ce None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:11.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed5d940-1ed9-4cb6-8b1d-66a2aeed33ce None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:11.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed5d940-1ed9-4cb6-8b1d-66a2aeed33ce None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:11.391674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 592/1167] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:11.396937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4de7d9e-a270-40e1-97e6-48056ab6c3af None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:11.397356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4de7d9e-a270-40e1-97e6-48056ab6c3af None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:11.397491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4de7d9e-a270-40e1-97e6-48056ab6c3af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:11.413783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f4de7d9e-a270-40e1-97e6-48056ab6c3af None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:11.414947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f4de7d9e-a270-40e1-97e6-48056ab6c3af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:11.479161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 576/1168] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:11 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:11.502900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed5d940-1ed9-4cb6-8b1d-66a2aeed33ce None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:11.545630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:11.603275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed5d940-1ed9-4cb6-8b1d-66a2aeed33ce None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:11.636954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9870d28-d8ac-451e-8dc4-0da52f23ee9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:11.637597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9870d28-d8ac-451e-8dc4-0da52f23ee9f None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:11.637883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9870d28-d8ac-451e-8dc4-0da52f23ee9f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:11.638164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9870d28-d8ac-451e-8dc4-0da52f23ee9f None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:11.638909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9870d28-d8ac-451e-8dc4-0da52f23ee9f None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:11.640311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9870d28-d8ac-451e-8dc4-0da52f23ee9f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:11.654000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 593/1169] 10.222.0.33 () {60 vars in 1138 bytes} [Tue Jul 19 10:53:11 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:53:13.685620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1986c2f2-bbee-4b51-9534-f73f924e894d None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:13.686147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1986c2f2-bbee-4b51-9534-f73f924e894d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:13.686147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1986c2f2-bbee-4b51-9534-f73f924e894d None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:13.687536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 577/1170] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:13.692576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:13.693044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:13.693151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:13.709966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:13.711143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:13.774976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 594/1171] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:13 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:13.798102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1986c2f2-bbee-4b51-9534-f73f924e894d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:13.826969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:13.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1986c2f2-bbee-4b51-9534-f73f924e894d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:13.928434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63861736-f555-47cf-a8ad-c406457a4dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:13.929431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63861736-f555-47cf-a8ad-c406457a4dc5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:13.929711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63861736-f555-47cf-a8ad-c406457a4dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:13.929982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63861736-f555-47cf-a8ad-c406457a4dc5 None admin] PATH_INFO: `/v3/domains/4223da3309ec487c84c18011f4f7a4c3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:13.940301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63861736-f555-47cf-a8ad-c406457a4dc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=4223da3309ec487c84c18011f4f7a4c3)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:13.941702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63861736-f555-47cf-a8ad-c406457a4dc5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:13.943313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 578/1172] 10.222.0.33 () {56 vars in 1183 bytes} [Tue Jul 19 10:53:13 2022] GET /identity/v3/domains/4223da3309ec487c84c18011f4f7a4c3 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:13.949060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:13.978626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:14.046549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bd7546-1968-4d8c-bab1-5cd73426d7a1 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:14.079474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dbdad4-6c14-43da-b82b-069bcddca809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:14.080115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbdad4-6c14-43da-b82b-069bcddca809 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:14.080397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbdad4-6c14-43da-b82b-069bcddca809 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:14.080670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbdad4-6c14-43da-b82b-069bcddca809 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:14.081408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dbdad4-6c14-43da-b82b-069bcddca809 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:14.082815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dbdad4-6c14-43da-b82b-069bcddca809 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:14.106486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 595/1173] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:53:13 2022] POST /identity/v3/projects => generated 361 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:53:16.124550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ffb1f3-5751-460a-93ff-faab7bebba53 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:16.125663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ffb1f3-5751-460a-93ff-faab7bebba53 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:16.125663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ffb1f3-5751-460a-93ff-faab7bebba53 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:16.127038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 579/1174] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:16.132290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:16.132671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:16.132883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:16.149222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:16.150400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:16.215047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 596/1175] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:16 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:16.240515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ffb1f3-5751-460a-93ff-faab7bebba53 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:16.328242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:16.338713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ffb1f3-5751-460a-93ff-faab7bebba53 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:16.371337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e907c781-2ead-4095-ae67-d4c0523d4da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:16.372170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907c781-2ead-4095-ae67-d4c0523d4da8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:16.372448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907c781-2ead-4095-ae67-d4c0523d4da8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:16.372744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907c781-2ead-4095-ae67-d4c0523d4da8 None admin] PATH_INFO: `/v3/domains/4223da3309ec487c84c18011f4f7a4c3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:16.374015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e907c781-2ead-4095-ae67-d4c0523d4da8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=4223da3309ec487c84c18011f4f7a4c3)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:16.375449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e907c781-2ead-4095-ae67-d4c0523d4da8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:16.377055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 580/1176] 10.222.0.33 () {56 vars in 1183 bytes} [Tue Jul 19 10:53:16 2022] GET /identity/v3/domains/4223da3309ec487c84c18011f4f7a4c3 => generated 267 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:16.382601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:16.396266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:16.479001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd67ae3-18ee-4ae6-a22d-85760d705173 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:16.511399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb5b3a4-c4cc-4c8b-bd34-4db165e51120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:16.512042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb5b3a4-c4cc-4c8b-bd34-4db165e51120 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:16.512319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb5b3a4-c4cc-4c8b-bd34-4db165e51120 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:16.512588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb5b3a4-c4cc-4c8b-bd34-4db165e51120 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:16.513306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb5b3a4-c4cc-4c8b-bd34-4db165e51120 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:16.514708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb5b3a4-c4cc-4c8b-bd34-4db165e51120 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:16.534186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 597/1177] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:53:16 2022] POST /identity/v3/users => generated 311 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:53:18.578809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8faac-68c3-4a00-887c-3d75c1f1f742 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:18.579355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8faac-68c3-4a00-887c-3d75c1f1f742 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:18.579355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8faac-68c3-4a00-887c-3d75c1f1f742 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:18.580722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 581/1178] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:18.585941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:18.586353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:18.586492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:18.603362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:18.604536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:18.664178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:18.670054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 598/1179] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:18 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:18.693504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8faac-68c3-4a00-887c-3d75c1f1f742 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:18.792673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8faac-68c3-4a00-887c-3d75c1f1f742 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:18.824556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:18.825343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:18.825623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:18.825894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:18.836087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:53:18.844109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:53:18.845213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:53:18.846246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 582/1180] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:53:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:53:18.851139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:18.946570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e205ef-7656-48c7-ab5d-c048a129e06a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:18.979621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:18.980261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:18.980544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:18.980849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:18.981589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:18.981894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:18.983257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:18.991392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 599/1181] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:53:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:18.997241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:19.015960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:19.094206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50d8f3-ee29-4c2a-9367-e809faf6dab5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:19.126286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:19.127127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:19.127406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:19.127676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] PATH_INFO: `/v3/users/614a4d1dbcb4440daf701bae7e2abacb` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:19.140115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] RBAC: Authorizing `identity:get_user(user_id=614a4d1dbcb4440daf701bae7e2abacb)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:19.142057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:19.143635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 583/1182] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:53:18 2022] GET /identity/v3/users/614a4d1dbcb4440daf701bae7e2abacb => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:19.149246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:19.206165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:19.254673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e3bfd9-abc8-4f15-8eae-138ae79092ab None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:19.286959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:19.287600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:19.287880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:19.288152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:19.355382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:19.356770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:19.411262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:19.451039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 600/1183] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:53:19 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:53:19.457537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:19.531037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:19.553621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df37d49-b77f-4310-9712-1c00c1a038f9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:19.587269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e915a5-425c-43b9-9857-4433d577558c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:19.588119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e915a5-425c-43b9-9857-4433d577558c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:19.588398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e915a5-425c-43b9-9857-4433d577558c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:19.588670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e915a5-425c-43b9-9857-4433d577558c None admin] PATH_INFO: `/v3/projects/a9b64553959744539fa855b46d71954a` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:19.589863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e915a5-425c-43b9-9857-4433d577558c None admin] RBAC: Authorizing `identity:get_project(project_id=a9b64553959744539fa855b46d71954a)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:19.591280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e915a5-425c-43b9-9857-4433d577558c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:19.594714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 584/1184] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:53:19 2022] GET /identity/v3/projects/a9b64553959744539fa855b46d71954a => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:19.600898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:19.697774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c896f7b-b5c6-43b1-bf36-ab1b51c8fd5c None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:19.729868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:19.730498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:19.730782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:19.731055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:19.731874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=a9b64553959744539fa855b46d71954a, user.id=614a4d1dbcb4440daf701bae7e2abacb) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:19.732182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:19.733620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24118a0f-3b3d-4428-a04d-aaa47afdfbb5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:19.739578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 601/1185] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:53:19 2022] GET /identity/v3/role_assignments?user.id=614a4d1dbcb4440daf701bae7e2abacb&scope.project.id=a9b64553959744539fa855b46d71954a&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:21.728312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57512a-8809-4adc-8f3b-459b53dda6c9 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:21.728882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57512a-8809-4adc-8f3b-459b53dda6c9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:21.728882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57512a-8809-4adc-8f3b-459b53dda6c9 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:21.730269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 585/1186] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:21.735917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:21.736315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:21.736472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:21.753090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:21.754270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:21.770796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:21.818797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 602/1187] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:21 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:21.840476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57512a-8809-4adc-8f3b-459b53dda6c9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:21.938311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57512a-8809-4adc-8f3b-459b53dda6c9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:21.971405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:21.972271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:21.972557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:21.972862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:21.982727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:53:21.990461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:53:21.991597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:53:21.992873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 586/1188] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:53:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:53:21.997629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:22.092251 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52b7d3c-236f-4eef-a90e-cce475b8cfac None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:22.125181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:22.125827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:22.126112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:22.126388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:22.127125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:22.127430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:22.128811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:22.136557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 603/1189] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:53:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:22.142927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:22.146592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:22.240908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd5a6a5-2b53-4aaa-aaf0-1c2419a14b3c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:22.273617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:22.274460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:22.274744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:22.275020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] PATH_INFO: `/v3/users/614a4d1dbcb4440daf701bae7e2abacb` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:22.276184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] RBAC: Authorizing `identity:get_user(user_id=614a4d1dbcb4440daf701bae7e2abacb)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:22.277611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:22.279205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 587/1190] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:53:22 2022] GET /identity/v3/users/614a4d1dbcb4440daf701bae7e2abacb => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:22.284865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:22.302848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:22.382832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068194cf-6537-47d7-9b96-0bb8a14925f7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:22.415518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c60243e-251f-43bb-8823-55720a6b0008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:22.416355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c60243e-251f-43bb-8823-55720a6b0008 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:22.416641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c60243e-251f-43bb-8823-55720a6b0008 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:22.416946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c60243e-251f-43bb-8823-55720a6b0008 None admin] PATH_INFO: `/v3/projects/a9b64553959744539fa855b46d71954a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:22.418114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c60243e-251f-43bb-8823-55720a6b0008 None admin] RBAC: Authorizing `identity:get_project(project_id=a9b64553959744539fa855b46d71954a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:22.419533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c60243e-251f-43bb-8823-55720a6b0008 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:22.421247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 604/1191] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:53:22 2022] GET /identity/v3/projects/a9b64553959744539fa855b46d71954a => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:22.427001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:22.494198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:22.523494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6723d14d-827e-475a-b882-2b62701dc4d0 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:22.555115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f57ce4-752e-4d5a-9d3f-03d9d6f4efc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:22.555981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f57ce4-752e-4d5a-9d3f-03d9d6f4efc0 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:22.556263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f57ce4-752e-4d5a-9d3f-03d9d6f4efc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:22.556541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f57ce4-752e-4d5a-9d3f-03d9d6f4efc0 None admin] PATH_INFO: `/v3/projects/a9b64553959744539fa855b46d71954a/users/614a4d1dbcb4440daf701bae7e2abacb/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:22.558512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f57ce4-752e-4d5a-9d3f-03d9d6f4efc0 None admin] RBAC: Authorizing `identity:create_grant(project_id=a9b64553959744539fa855b46d71954a, user_id=614a4d1dbcb4440daf701bae7e2abacb, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:22.559921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f57ce4-752e-4d5a-9d3f-03d9d6f4efc0 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:22.572549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 588/1192] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:53:22 2022] PUT /identity/v3/projects/a9b64553959744539fa855b46d71954a/users/614a4d1dbcb4440daf701bae7e2abacb/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:53:24.565488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875e867-7cd4-4511-8ac5-70b57ebe2e1f None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:24.566039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875e867-7cd4-4511-8ac5-70b57ebe2e1f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:24.566039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875e867-7cd4-4511-8ac5-70b57ebe2e1f None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:24.567421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 605/1193] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:53:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:53:24.572665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:24.573111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:24.573256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:24.590416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:53:24.591603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:53:24.655657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 589/1194] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:53:24 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:53:24.679839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875e867-7cd4-4511-8ac5-70b57ebe2e1f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:24.762536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:24.779047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875e867-7cd4-4511-8ac5-70b57ebe2e1f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:24.812623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:24.813420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:24.813706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:24.813982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:24.823926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:53:24.831467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:53:24.832540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:53:24.833442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 606/1195] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:53:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:53:24.838451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:24.897757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:24.936012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec2ff2d-b444-4744-ac18-d3d9de70a652 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:24.969191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:24.969841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:24.970120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:24.970387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:24.971126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:24.971425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:24.972791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:24.980480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 590/1196] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:53:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:53:24.986280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:25.082622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e7406d-94f8-4b78-89fe-ac02a33cc913 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:25.115104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:25.115941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:25.116221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:25.116493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] PATH_INFO: `/v3/users/614a4d1dbcb4440daf701bae7e2abacb` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:25.117685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] RBAC: Authorizing `identity:get_user(user_id=614a4d1dbcb4440daf701bae7e2abacb)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:25.119121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:25.120713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 607/1197] 10.222.0.33 () {56 vars in 1177 bytes} [Tue Jul 19 10:53:24 2022] GET /identity/v3/users/614a4d1dbcb4440daf701bae7e2abacb => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:25.126344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:25.222272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffbef84-9d40-4767-83b5-fa94c228f3ae None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:25.244102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:25.254999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:25.255630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:25.255910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:25.256180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:25.319973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:25.321495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:25.416417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 591/1198] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:53:25 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:53:25.422238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:25.436845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:25.521277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a787021-f9a4-4d4d-b6c9-8f73c63fc8b7 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:25.553457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cacd33-639f-4068-96df-c2a54ceeb414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:25.554300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cacd33-639f-4068-96df-c2a54ceeb414 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:25.554586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cacd33-639f-4068-96df-c2a54ceeb414 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:25.554862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cacd33-639f-4068-96df-c2a54ceeb414 None admin] PATH_INFO: `/v3/projects/a9b64553959744539fa855b46d71954a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:25.556029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cacd33-639f-4068-96df-c2a54ceeb414 None admin] RBAC: Authorizing `identity:get_project(project_id=a9b64553959744539fa855b46d71954a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:25.557482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cacd33-639f-4068-96df-c2a54ceeb414 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:25.559174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 608/1199] 10.222.0.33 () {56 vars in 1186 bytes} [Tue Jul 19 10:53:25 2022] GET /identity/v3/projects/a9b64553959744539fa855b46d71954a => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:25.565630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:53:25.589554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:25.669137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb14911-279f-4824-acaa-2d248a32963c None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:53:25.702319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:53:25.702987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:53:25.703273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:53:25.703548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:53:25.704370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=a9b64553959744539fa855b46d71954a, user.id=614a4d1dbcb4440daf701bae7e2abacb) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:53:25.704686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:53:25.706129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69350088-eeef-4bdd-8525-e36d574b3603 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:53:25.711790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 592/1200] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:53:25 2022] GET /identity/v3/role_assignments?user.id=614a4d1dbcb4440daf701bae7e2abacb&scope.project.id=a9b64553959744539fa855b46d71954a&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:53:35.549352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=60 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:53:35.709162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=56 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:20.989577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ea051c-d8d7-44e3-978c-e89b422cb3ad None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:20.990434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ea051c-d8d7-44e3-978c-e89b422cb3ad None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:20.990658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ea051c-d8d7-44e3-978c-e89b422cb3ad None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:20.992771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 609/1201] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:54:20.998212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:20.998666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:20.998746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:21.020501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:21.021741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:21.139614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 593/1202] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:20 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:21.181713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ea051c-d8d7-44e3-978c-e89b422cb3ad None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:21.320480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ea051c-d8d7-44e3-978c-e89b422cb3ad None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:21.366473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:21.367834 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:21.368262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:21.368540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:21.386747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:21.388665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:21.411679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:21.413007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:21.414003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:21.414003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:21.414003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:21.414003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:21.414003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 610/1203] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:21.419175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:21.557475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf6978-8f1e-4f82-a8eb-2505e9bda0b0 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:21.599321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:21.612882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:21.612882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:21.612882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:21.612882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:21.613349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:21.613349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:21.614423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d5be8b-199f-4b53-8a93-ad802dbd8b43 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:21.631866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 594/1204] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:21.634334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:21.712574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:21.770405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9810e601-ea4d-4104-a3b0-ceba140e3de8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:21.819665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4176cf9b-8727-4889-912e-220abb108774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:21.820458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4176cf9b-8727-4889-912e-220abb108774 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:21.820801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4176cf9b-8727-4889-912e-220abb108774 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:21.821165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4176cf9b-8727-4889-912e-220abb108774 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:21.822010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4176cf9b-8727-4889-912e-220abb108774 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:21.823457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4176cf9b-8727-4889-912e-220abb108774 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:21.857866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 611/1205] 10.222.0.33 () {60 vars in 1132 bytes} [Tue Jul 19 10:54:21 2022] POST /identity/v3/users => generated 251 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 10:54:24.116409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f340d3-5a7c-429e-b74f-2102d667b062 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:24.118603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f340d3-5a7c-429e-b74f-2102d667b062 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:24.118603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f340d3-5a7c-429e-b74f-2102d667b062 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:24.118603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 595/1206] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:24.124601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:24.125113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:24.125326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:24.142382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:24.143568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:24.217082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 612/1207] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:24 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:24.240782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f340d3-5a7c-429e-b74f-2102d667b062 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:24.349561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f340d3-5a7c-429e-b74f-2102d667b062 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:24.386598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:24.387399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:24.387688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:24.387997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:24.402248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:54:24.410625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:24.411587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:24.412426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 596/1208] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:54:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:54:24.417922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:24.459331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:24.532409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f30e5a-6148-46de-b96d-7bc7df7f1991 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:24.572112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:24.572813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:24.573098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:24.573374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:24.574145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:24.574447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:24.575796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:24.587477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 613/1209] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:54:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:54:24.589883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:24.645489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:24.705641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12ed31-fd6e-4ae1-973e-40cbf68ef0e3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:24.741963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:24.742751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:24.743029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:24.743300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:24.754292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:24.755773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:24.774496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:24.775550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:24.776552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:24.776552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:24.776552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:24.776552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:24.776552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 597/1210] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:24.782269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:24.892718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:24.895719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89c528a-161b-4a96-ae65-96e96ac33280 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:24.941071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:24.942086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:24.942574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:24.943070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:24.944030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:24.944535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:24.946034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:24.963087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 614/1211] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:24.969481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:25.090820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1aa07b-607b-4b00-8c49-31b3475bbd8a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:25.092650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:25.132766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:25.132766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:25.132766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:25.133582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] PATH_INFO: `/v3/users/placement` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:25.140789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:25.143144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:54:25.158619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:25.159658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:25.160634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 598/1212] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:54:25.165731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:25.281680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1aefbf-92aa-4c51-b03e-190fc636693e None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:25.319646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:25.321929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:25.322224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:25.322512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:25.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:25.323640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:25.327199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:25.350798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 615/1213] 10.222.0.33 () {56 vars in 1143 bytes} [Tue Jul 19 10:54:25 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:54:25.352206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:25.466624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cce0b3-285a-4809-bf7d-0ddb8af8824a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:25.502788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:25.503568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:25.504624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:25.504624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:25.537995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:25.580161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:25.581713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:25.704120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 599/1214] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:54:25 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:54:25.710631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:25.732782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:25.826630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fbad82-cea2-4e77-8130-063f176fd169 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:25.866759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:25.867582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:25.867832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:25.868136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:25.878687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:25.880139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:25.898521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:25.899644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:25.900575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:25.900575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:25.900575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:25.900575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:25.900575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 616/1215] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:25.905952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:25.974463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:26.017663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73756c2e-fc68-4081-a0c2-7698dbf6aeb3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:26.052660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:26.053511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:26.055213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:26.055577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:26.056429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:26.056866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:26.058322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:26.072735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 600/1216] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:54:26.076306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:26.156255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:26.195728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74fc9ac-3602-420c-92e9-24884eb1659a None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:26.233260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:26.234103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:26.234384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:26.234656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:26.241595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:26.243020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:54:26.257915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:26.259042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:26.260101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 617/1217] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:54:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:54:26.266070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:26.376285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826dcd0a-7374-4ecf-ac58-6f9ff43af5f2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:26.401247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:26.416303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:26.417182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:26.417527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:26.417949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:26.418856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:26.419294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:26.421510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afcbc00-5c9b-40de-9713-bee2afa2ba1c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:26.435059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 601/1218] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:54:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:26.441807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:26.554849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9126798d-870d-424c-a13c-0648026517a0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:26.586015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:26.593481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:26.594313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:26.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:26.595236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:26.596077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=8234f48f37ae4384bf6c5f81c664bf9b) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:26.596389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:26.598020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82250d3-91a6-4227-8173-b2cb218217c5 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:26.604773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 618/1219] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:54:26 2022] GET /identity/v3/role_assignments?user.id=8234f48f37ae4384bf6c5f81c664bf9b&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:28.857502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cca0d-0bb5-4a75-b85b-ba5e3d566ad9 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:28.858033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cca0d-0bb5-4a75-b85b-ba5e3d566ad9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:28.858033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cca0d-0bb5-4a75-b85b-ba5e3d566ad9 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:28.859376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 602/1220] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:28.865082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:28.865748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:28.865884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:28.883641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:28.885072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:28.962117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 619/1221] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:28 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:28.984649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5cca0d-0bb5-4a75-b85b-ba5e3d566ad9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:29.112978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5cca0d-0bb5-4a75-b85b-ba5e3d566ad9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:29.147591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:29.148369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:29.148650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:29.148948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:29.160021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:54:29.169085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:29.170060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:29.171531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 603/1222] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:54:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:29.177258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:29.279634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d116ae60-048e-47e9-bcf4-a5808164edeb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:29.304449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:29.315130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:29.315792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:29.316074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:29.316349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:29.317148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] RBAC: Adding query filter params (name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:29.317455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:29.318807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:29.331137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 620/1223] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:54:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:29.332354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:29.383004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:29.439163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dc120f-83ae-4dc0-9d98-03681e796ef9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:29.476247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:29.477074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:29.477359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:29.477633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:29.488659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:29.490383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:29.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:29.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:29.510718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:29.510718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:29.510718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:29.510718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:29.510718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 604/1224] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:29.516326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:29.621269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5790f8ed-036f-42a3-9a10-3f7a69a847c8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:29.660606 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:29.661426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:29.661814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:29.662098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:29.662883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:29.663169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:29.664607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:29.677728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 621/1225] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:29.684430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:29.785054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:29.792478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa333c0-b70e-425d-b3da-0d97ef3f80ba None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:29.826932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:29.828510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:29.828948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:29.829363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:29.837296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:29.838946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:54:29.853980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:29.855173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:29.856326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 605/1226] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:29 2022] GET /identity/v3/users/placement => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:29.861548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:29.913970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:29.973120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec24612-2b68-4300-8f6b-877b511e1f6f None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:30.016367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:30.017107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:30.017394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:30.017740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:30.018639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:30.018954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:30.020372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:30.039026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 622/1227] 10.222.0.33 () {56 vars in 1143 bytes} [Tue Jul 19 10:54:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:30.045515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:30.160589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9350d-379b-485b-8558-c5b17e0c3822 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:30.196825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:30.197621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:30.197898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:30.198172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:30.211016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:30.212455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:30.230897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:30.232062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:30.232986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:30.232986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:30.232986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:30.232986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:30.232986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 606/1228] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:30.238485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:30.346654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69d88f6-26a8-4214-a9cf-dc9b32ab07c2 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:30.348985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:30.384503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:30.385334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:30.385651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:30.385984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:30.386813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:30.387144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:30.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:30.402542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 623/1229] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:30.409127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:30.419914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:30.522654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3ff66a-9b27-4232-a0e7-b28b7fc68f5a None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:30.560061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:30.560881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:30.561164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:30.561462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:30.569925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:30.571375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:54:30.586733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:30.587819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:30.588796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 607/1230] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:54:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:30.594595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:30.711384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad603b02-b015-40d8-a829-82fe36b3946b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:30.752957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:30.753612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:30.753892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:30.754165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:30.754937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:30.755242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:30.756645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083f4bf9-3bcc-495b-98f7-ac4e0a85057a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:30.770286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 624/1231] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:54:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:30.776408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:30.853102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:30.897617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd72cc7-9c7a-4a17-bcc3-e8ca74999121 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:30.934592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eba491b-5d85-40bc-a41e-8bf0605c72d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:30.935306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eba491b-5d85-40bc-a41e-8bf0605c72d5 None admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:30.935586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eba491b-5d85-40bc-a41e-8bf0605c72d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:30.935855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eba491b-5d85-40bc-a41e-8bf0605c72d5 None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/8234f48f37ae4384bf6c5f81c664bf9b/roles/a0ef2bc7414d4eee9465dcc1abe01b32` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:30.951886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eba491b-5d85-40bc-a41e-8bf0605c72d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=8234f48f37ae4384bf6c5f81c664bf9b, role_id=a0ef2bc7414d4eee9465dcc1abe01b32)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:30.953853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eba491b-5d85-40bc-a41e-8bf0605c72d5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:30.966474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 608/1232] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:54:30 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/8234f48f37ae4384bf6c5f81c664bf9b/roles/a0ef2bc7414d4eee9465dcc1abe01b32 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:54:33.112062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689dbec-4c98-4294-8e10-0c6dd6530150 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:33.112579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689dbec-4c98-4294-8e10-0c6dd6530150 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:33.112713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689dbec-4c98-4294-8e10-0c6dd6530150 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:33.114078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 625/1233] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:33.119514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:33.119950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:33.120041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:33.137266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:33.138655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:33.210333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 609/1234] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:33 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:33.233696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689dbec-4c98-4294-8e10-0c6dd6530150 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:33.259289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:33.345485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689dbec-4c98-4294-8e10-0c6dd6530150 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:33.382563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:33.383387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:33.383698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:33.384010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] PATH_INFO: `/v3/roles/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:33.394646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:54:33.402848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:33.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:33.405217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 626/1235] 10.222.0.33 () {56 vars in 1102 bytes} [Tue Jul 19 10:54:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:33.412726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:33.474508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:33.538507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dae12ad-5182-41d4-a542-1e5478ee9660 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:33.574100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:33.574762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:33.575043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:33.575318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:33.576062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] RBAC: Adding query filter params (name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:33.576366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:33.577862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:33.587205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 610/1236] 10.222.0.33 () {56 vars in 1103 bytes} [Tue Jul 19 10:54:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:33.593568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:33.689937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:33.707071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2125a1c-867a-46c7-823e-743c15ff16dc None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:33.748912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:33.749597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:33.749920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:33.750184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:33.763132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:33.768993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:33.793449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:33.794668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:33.795647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:33.795647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:33.795647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:33.795647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:33.795647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 627/1237] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:33.801545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:33.915488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d86f094-6766-4e15-b565-f1fec80a2885 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:33.926510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:33.955230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:33.955882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:33.956165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:33.956438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:33.957191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:33.957497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:33.958858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:33.970758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 611/1238] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:33.976718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:34.115813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41965915-21ab-488c-ac74-a600f9549067 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:34.150135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:34.150923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:34.151205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:34.151473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] PATH_INFO: `/v3/users/placement` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:34.154200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:34.161254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:34.162934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:54:34.179218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:34.180486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:34.181476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 628/1239] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:33 2022] GET /identity/v3/users/placement => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:34.186598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:34.296522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb14aa7-33a3-41b2-98c7-9ec0822346cc None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:34.336240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:34.337115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:34.337515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:34.337909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:34.338867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:34.339191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:34.340617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:34.362321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 612/1240] 10.222.0.33 () {56 vars in 1143 bytes} [Tue Jul 19 10:54:34 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:54:34.368862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:34.483915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e5a53-1327-4920-9baa-b3780bd97eaf None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:34.519240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:34.519978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:34.520343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:34.520622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:34.588335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:34.595341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:34.596987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:34.723023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 629/1241] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:54:34 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:54:34.724535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:34.755323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:34.834847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2472ab2-a6ca-4c47-8251-c122e0092396 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:34.871320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:34.872173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:34.872515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:34.872857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:34.885170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:34.886738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:34.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:34.906870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:34.907987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:34.907987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:34.907987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:34.907987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:34.907987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 613/1242] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:34.912841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:35.020728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:35.027209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db19e8-9658-475a-9a1b-1b1e2fcc6a97 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:35.063628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:35.064427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:35.064839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:35.065217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:35.066019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:35.066354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:35.067808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:35.079616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 630/1243] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:35.085330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:35.174210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:35.197903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4e8c2-a0f2-42e2-b59e-a5291d4eab89 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:35.232775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:35.234180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:35.234465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:35.234740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:35.241548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:35.242987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:54:35.256674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:35.257739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:35.258784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 614/1244] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:54:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:35.263839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:35.376927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bf25f1-f899-4cb3-8c2f-d034e994e698 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:35.414218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:35.414858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:35.415155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:35.415466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:35.416437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:35.416810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:35.418457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d887514-d5c2-415b-a940-c1204b93d9fb None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:35.432434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 631/1245] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:54:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:35.438552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:35.550778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4477db-de58-451b-82e0-a7035299e3fa None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:35.586710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:35.587524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:35.587842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:35.588226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:35.589249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] RBAC: Adding query filter params (role.id=a0ef2bc7414d4eee9465dcc1abe01b32, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=8234f48f37ae4384bf6c5f81c664bf9b) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:35.589563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:35.591337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8ca25f-e9fc-4d4a-bc52-75c4a35a85a8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:35.599051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 615/1246] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:54:35 2022] GET /identity/v3/role_assignments?user.id=8234f48f37ae4384bf6c5f81c664bf9b&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=a0ef2bc7414d4eee9465dcc1abe01b32 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:37.708782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6b689-d56b-4321-8f2d-e7e1ae8e0be6 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:37.709320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6b689-d56b-4321-8f2d-e7e1ae8e0be6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:37.709320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6b689-d56b-4321-8f2d-e7e1ae8e0be6 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:37.710706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 632/1247] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:37.716553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:37.717193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:37.717362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:37.723156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:37.739396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:37.741010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:37.832721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 616/1248] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:37 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:37.846924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6b689-d56b-4321-8f2d-e7e1ae8e0be6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:37.875153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:37.970286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6b689-d56b-4321-8f2d-e7e1ae8e0be6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:38.009768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:38.010636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:38.010955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:38.011228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:38.022647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:54:38.032074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:38.033314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:38.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 633/1249] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:54:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:38.040021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:38.151103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5642b47-59dd-41b2-adee-0aa49bcc9fde None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:38.184202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:38.189616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:38.190302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:38.190929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:38.191271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:38.192000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:38.192310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:38.193719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:38.202638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 617/1250] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:54:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:38.208975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:38.301921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:38.327920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb46d700-ec2f-4ccc-a1c1-b4dd7dca58f8 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:38.368634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:38.369463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:38.369753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:38.370079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:38.381911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:38.383513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:38.402323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:38.404414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:38.406829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:38.406829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:38.406829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:38.406829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:38.406829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 634/1251] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 19 10:54:38.410949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:38.541487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add4ade0-95cd-4706-86bb-51bd7be4d490 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:38.579388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:38.579992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:38.580263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:38.580532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:38.581296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:38.581602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:38.582959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:38.596504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 618/1252] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:38.602730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:38.730976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aed9532-23d2-4077-acaa-e833d0d03c0b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:38.758925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:38.767926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:38.768716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:38.769036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:38.769294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:38.777554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:38.779197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:54:38.793838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:38.794948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:38.796035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 635/1253] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:38 2022] GET /identity/v3/users/placement => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:38.801238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:38.861220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:38.908879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a01bee-1600-47f8-bf5a-9386d6c65d75 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:38.947441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:38.948110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:38.948395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:38.948667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:38.949531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:38.949840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:38.951276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:38.972136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 619/1254] 10.222.0.33 () {56 vars in 1143 bytes} [Tue Jul 19 10:54:38 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:38.978132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:39.094933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c0ab6-ca8b-4b43-9128-0da4f3d5a170 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:39.130422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:39.130991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:39.131275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:39.131545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:39.194157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:39.209083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:39.210527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:39.320138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 636/1255] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:54:38 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:54:39.326576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:39.432052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf37258-46a4-41fc-9102-289e1ecf4bbe None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:39.448339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:39.474920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:39.475934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:39.476349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:39.476634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:39.488824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:39.490525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:39.508682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:39.509590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:39.510489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:39.510489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:39.510489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:39.510489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:39.510489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 620/1256] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:39.515804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:39.620267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:39.633380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580e950-fd5b-473e-9858-99988f04e564 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:39.675096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:39.675863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:39.676267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:39.676600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:39.677515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:39.677899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:39.679584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:39.694105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 637/1257] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:39.700607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:39.808317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050189d4-6900-4433-95a0-ee928c413b14 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:39.850951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:39.851762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:39.852044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:39.852315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:39.859866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:39.861394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:54:39.876136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:39.877339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:39.878593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 621/1258] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:54:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:39.883729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:39.994308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af90a34-95c9-436a-ab09-1103f53ef226 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:40.031899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:40.032614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:40.032987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:40.033290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:40.034189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:40.034505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:40.036111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d766995f-f6cd-468f-a26d-8a2be5ebd847 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:40.051528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 638/1259] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:54:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 10:54:40.056456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:40.085794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:40.170415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c06413-007d-4604-8db2-347b55e09b41 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:40.207750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:40.208424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:40.208736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:40.209008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:40.209861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=8234f48f37ae4384bf6c5f81c664bf9b) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:40.210173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:40.211635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8101dd-3f4c-4488-a455-06e384dcea4c None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:40.218647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 622/1260] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:54:40 2022] GET /identity/v3/role_assignments?user.id=8234f48f37ae4384bf6c5f81c664bf9b&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=90dd7541fe26472fb3843454a738e154 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:42.239730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f87d8b3-8f0e-4cc0-8baa-b01d080ff4b6 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:42.240237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f87d8b3-8f0e-4cc0-8baa-b01d080ff4b6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:42.240237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f87d8b3-8f0e-4cc0-8baa-b01d080ff4b6 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:42.241840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 639/1261] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:42.246848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:42.247263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:42.247411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:42.265535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:42.266725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:42.331240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 623/1262] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:42.352716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f87d8b3-8f0e-4cc0-8baa-b01d080ff4b6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:42.363782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:42.451229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f87d8b3-8f0e-4cc0-8baa-b01d080ff4b6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:42.483926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:42.484741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:42.485024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:42.485301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:42.495248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:54:42.502822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:42.503887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:42.504962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 640/1263] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:54:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:42.509707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:42.527498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:42.605994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fb6174-dba2-4775-b510-ba2a7356223a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:42.638872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:42.639573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:42.639813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:42.640085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:42.640935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:42.641229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:42.642608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:42.651363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 624/1264] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:54:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:42.659425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:42.732920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:42.776099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d65e0a-fbfe-4ae9-a432-32ab03888ca0 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:42.808673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:42.809471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:42.809752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:42.810023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:42.820113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:42.821569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:42.837820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:42.838843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:42.839807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:42.839807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:42.839807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:42.839807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:42.839807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 641/1265] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:42.845474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:42.924713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:42.943615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810d485-40fd-44ac-800f-005693fa83c8 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:42.976639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:42.977317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:42.977602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:42.977877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:42.978598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:42.978901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:42.980257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:42.991872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 625/1266] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:42.997621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:43.094032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a85872-8dd4-473c-a1db-c0c9c3c4159b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:43.112648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:43.126550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:43.127333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:43.127612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:43.127882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] PATH_INFO: `/v3/users/placement` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:43.135739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:43.137224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:54:43.151534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:43.152573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:43.153572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 642/1267] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:42 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:43.159286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:43.255383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1198a0-21ff-4f46-a333-c288a2cd1679 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:43.288758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:43.289413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:43.289686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:43.289959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:43.290784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:43.291088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:43.292495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:43.300857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:43.309353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 626/1268] 10.222.0.33 () {56 vars in 1143 bytes} [Tue Jul 19 10:54:43 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:43.315355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:43.410972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda055d-7ea4-4b12-9c5a-0784c7c8e60b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:43.442763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:43.443519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:43.443798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:43.444068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:43.454666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:43.456079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:43.472397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:43.473421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:43.474503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:43.474503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:43.474503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:43.474503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:43.474503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 643/1269] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:43.479037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:43.575097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53290e26-f14b-4d73-9bed-08da2fd2ead5 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:43.607730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:43.608914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:43.609197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:43.609468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:43.610181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:43.610483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:43.611821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:43.622893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 627/1270] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:43.628781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:43.661996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:43.732234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e1185-34d3-490e-a4df-a2dcab2e932d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:43.765813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:43.766589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:43.766873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:43.767146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:43.773597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:43.775028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:54:43.788872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:43.789875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:43.790949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 644/1271] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:54:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:43.796469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:43.852571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:43.894482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffa3348-7c96-40c9-9e14-2cdd383c7bae None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:43.927009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:43.927660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:43.927946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:43.928222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:43.929062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:43.929384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:43.930816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c7ce4-c82b-4c6c-9fc4-11d224ef992f None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:43.943336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 628/1272] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:54:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:43.949377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:44.037655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:44.046582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e671d-3ff8-42e1-8a2b-f44337e8e545 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:44.079607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8e63dc-3caa-4487-8140-4a3e050fa20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:44.080467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8e63dc-3caa-4487-8140-4a3e050fa20b None admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:44.080774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8e63dc-3caa-4487-8140-4a3e050fa20b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:44.081053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8e63dc-3caa-4487-8140-4a3e050fa20b None admin] PATH_INFO: `/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/8234f48f37ae4384bf6c5f81c664bf9b/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:44.082955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8e63dc-3caa-4487-8140-4a3e050fa20b None admin] RBAC: Authorizing `identity:create_grant(project_id=528567e0c4c74f9ba0ff9a463cfc0218, user_id=8234f48f37ae4384bf6c5f81c664bf9b, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:44.084365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8e63dc-3caa-4487-8140-4a3e050fa20b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:44.096396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 645/1273] 10.222.0.33 () {58 vars in 1439 bytes} [Tue Jul 19 10:54:43 2022] PUT /identity/v3/projects/528567e0c4c74f9ba0ff9a463cfc0218/users/8234f48f37ae4384bf6c5f81c664bf9b/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 10:54:46.075410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf4c55-df0f-489a-bf0f-4c6f456c4ac1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.075952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf4c55-df0f-489a-bf0f-4c6f456c4ac1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.075952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf4c55-df0f-489a-bf0f-4c6f456c4ac1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.077863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 629/1274] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:46.082982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.083388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.083530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.100242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:46.101438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:46.164211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 646/1275] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:46 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:46.188176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcf4c55-df0f-489a-bf0f-4c6f456c4ac1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:46.286876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcf4c55-df0f-489a-bf0f-4c6f456c4ac1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:46.303872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:46.320193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:46.321001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.321288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.321563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.331323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 19 10:54:46.338901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:46.339809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:46.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 630/1276] 10.222.0.33 () {56 vars in 1096 bytes} [Tue Jul 19 10:54:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:46.346524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:46.424193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:46.442711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cb70e8-1982-4489-8ba1-f00cd92c8a7a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:46.475847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:46.476493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.476802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.477077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.477812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:46.478114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:46.479470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:46.487276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 647/1277] 10.222.0.33 () {56 vars in 1099 bytes} [Tue Jul 19 10:54:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:46.493007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:46.589238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd9ceb-5dcc-4501-bf7b-111a022b3910 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:46.621716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:46.622479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.622762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.623037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.633755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:46.635180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:46.653258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:46.654476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:46.655441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:46.655441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:46.655441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:46.655441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:46.655441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 631/1278] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:46.661373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:46.761947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8a58fd-f45e-49d8-893d-b8ebbd4e759d None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:46.793473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:46.794121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.794405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.794680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.795397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:46.796188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:46.797572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:46.809053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 648/1279] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:46.815138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:46.845415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:46.913654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8247fa33-150b-450d-af4e-2f1273e518d2 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:46.947056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:46.947851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:46.948135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:46.948408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] PATH_INFO: `/v3/users/placement` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:46.955791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:46.957258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 19 10:54:46.971722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:46.972815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:46.973957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 632/1280] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:46 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:46.978804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:46.982542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:47.074130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bb709f-e19c-45f6-8dad-b619b6c5ae7b None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:47.106211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:47.106855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:47.107131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:47.107400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:47.108846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:47.109150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:47.110545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:47.127761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 649/1281] 10.222.0.33 () {56 vars in 1143 bytes} [Tue Jul 19 10:54:46 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:47.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:47.220642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:47.230268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4c65d1-1b4f-440d-a8fa-3f4b0926fb3e None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:47.262515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:47.263134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:47.263415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:47.263684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:47.327297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:47.328654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:47.412926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:47.423086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 633/1282] 10.222.0.33 () {58 vars in 1282 bytes} [Tue Jul 19 10:54:47 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 19 10:54:47.428768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:47.490086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:47.527160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d44226a-9eb9-4edc-a23b-ca59fd45df68 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:47.559175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:47.559943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:47.560221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:47.560491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:47.570093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:47.571512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 19 10:54:47.588825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:47.589849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:47.591174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 19 10:54:47.591174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 19 10:54:47.591174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 19 10:54:47.591174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:47.591174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 650/1283] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:54:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:47.596187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:47.694825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026783df-f4c6-4c6f-adb6-4956dbf63147 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:47.727798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:47.728439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:47.728739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:47.729014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:47.729733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:47.730036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:47.731380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:47.742421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 634/1284] 10.222.0.33 () {56 vars in 1109 bytes} [Tue Jul 19 10:54:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:47.748144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:47.843806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d021733e-e6b0-487b-bd3b-8ca463f45875 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:47.856909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:47.877039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:47.877797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:47.878076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:47.878343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:47.884653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:47.886107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 19 10:54:47.899788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:47.900836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: ERROR keystone.server.flask.application Jul 19 10:54:47.901816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 651/1285] 10.222.0.33 () {56 vars in 1111 bytes} [Tue Jul 19 10:54:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:47.906801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:47.956136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:48.004530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933cc7f-2a5d-4e07-b972-cb87384b7ce9 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:48.036382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:48.037051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:48.037359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:48.037630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:48.038405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:48.038708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:48.040110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b03de2-549d-4dbc-a87f-e5b78521f474 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:48.052202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 635/1286] 10.222.0.33 () {56 vars in 1148 bytes} [Tue Jul 19 10:54:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:48.058160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:48.155038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32456c29-6382-461a-97a5-88089c5a7777 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:48.187691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:48.188313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:48.188592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:48.188888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:48.189705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] RBAC: Adding query filter params (role.id=90dd7541fe26472fb3843454a738e154, scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218, user.id=8234f48f37ae4384bf6c5f81c664bf9b) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:48.190011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:48.191414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f57a5d2-6630-41f4-bd90-74da1ca72484 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:48.197033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 652/1287] 10.222.0.33 () {56 vars in 1374 bytes} [Tue Jul 19 10:54:48 2022] GET /identity/v3/role_assignments?user.id=8234f48f37ae4384bf6c5f81c664bf9b&scope.project.id=528567e0c4c74f9ba0ff9a463cfc0218&role.id=90dd7541fe26472fb3843454a738e154 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:50.213748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d59d0c-4d83-44e2-b6b3-eae7a3c47f4f None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:50.214231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d59d0c-4d83-44e2-b6b3-eae7a3c47f4f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:50.214231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d59d0c-4d83-44e2-b6b3-eae7a3c47f4f None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:50.215572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 636/1288] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:50.221269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:50.221706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:50.221799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:50.238662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:50.239830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:50.267927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:50.304529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 653/1289] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:50.327300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d59d0c-4d83-44e2-b6b3-eae7a3c47f4f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:50.426441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d59d0c-4d83-44e2-b6b3-eae7a3c47f4f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:50.436707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:50.460940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:50.461725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:50.462006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:50.462278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:50.462923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:50.464311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:50.476448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:50.477646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:50.478683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 637/1290] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:54:50 2022] GET /identity/v3/services/placement => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:50.483830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:50.577216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0da2cf-28d7-40da-91a6-3c989a140ddb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:50.609435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:50.610068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:50.610346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:50.610613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:50.611304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:50.611604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:50.612964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7519cb5-d108-4c2b-91a0-74f769733554 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:50.617885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 654/1291] 10.222.0.33 () {56 vars in 1116 bytes} [Tue Jul 19 10:54:50 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:50.623452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:50.730465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab7b658-02fd-4168-bb6a-bdb8a61c9b77 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:50.762750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:50.763393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:50.763674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:50.763944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] PATH_INFO: `/v3/services` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:50.765302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:50.765611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:50.766949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfcb627-4d06-4a76-af51-0b6aab22a063 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:50.771237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 638/1292] 10.222.0.33 () {56 vars in 1116 bytes} [Tue Jul 19 10:54:50 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:52.761917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ec3d8-c5b3-4b12-90ff-8f3fd9a2a5df None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:52.762497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ec3d8-c5b3-4b12-90ff-8f3fd9a2a5df None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:52.762497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ec3d8-c5b3-4b12-90ff-8f3fd9a2a5df None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:52.763829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 655/1293] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:52.769142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbe094e-6f65-48c1-ab28-934cab9192d0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:52.769598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbe094e-6f65-48c1-ab28-934cab9192d0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:52.769705 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbe094e-6f65-48c1-ab28-934cab9192d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:52.786041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-0bbe094e-6f65-48c1-ab28-934cab9192d0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:52.787213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0bbe094e-6f65-48c1-ab28-934cab9192d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:52.816442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:52.852260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 639/1294] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:52.876041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4ec3d8-c5b3-4b12-90ff-8f3fd9a2a5df None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:52.906037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:52.974863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4ec3d8-c5b3-4b12-90ff-8f3fd9a2a5df None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:53.007713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e5bd06-8bae-424c-9767-730d16784779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:53.008359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5bd06-8bae-424c-9767-730d16784779 None admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:53.008641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5bd06-8bae-424c-9767-730d16784779 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:53.008944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5bd06-8bae-424c-9767-730d16784779 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:53.009668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5bd06-8bae-424c-9767-730d16784779 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:53.011013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5bd06-8bae-424c-9767-730d16784779 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:53.023336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 656/1295] 10.222.0.33 () {60 vars in 1141 bytes} [Tue Jul 19 10:54:52 2022] POST /identity/v3/services => generated 246 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:54:55.037356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880592bb-ca0b-458d-9f91-2511a9eddaed None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:55.037889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880592bb-ca0b-458d-9f91-2511a9eddaed None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:55.037889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880592bb-ca0b-458d-9f91-2511a9eddaed None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:55.039286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 640/1296] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:55.044294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:55.044679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:55.044883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:55.061113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:55.062277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:55.133339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 657/1297] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:55 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 19 10:54:55.156869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880592bb-ca0b-458d-9f91-2511a9eddaed None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:55.253961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880592bb-ca0b-458d-9f91-2511a9eddaed None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:55.286122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:55.286889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:55.287168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:55.287435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:55.288073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:55.289445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:55.301161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:55.302406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:55.303778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 641/1298] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:54:55 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:55.308175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:55.326171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:55.405129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282e9b81-fd24-4727-9a4f-806ed3e904fe None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:55.438834 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:55.439494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:55.439781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:55.440056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:55.440793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:55.441103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:55.442508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:55.447594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 658/1299] 10.222.0.33 () {56 vars in 1116 bytes} [Tue Jul 19 10:54:55 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:55.454209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:55.484890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:55.551403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507fdaae-dd4e-4cf6-9eb1-cd3d4ada4491 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:55.583874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:55.584506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:55.584828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:55.585102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:55.585878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] RBAC: Adding query filter params (interface=public, service_id=41cb3534e17f4a17940479fe870be63c, region_id=RegionOne) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:55.586185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:55.587521 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2823d0eb-67a9-412d-ae1f-cc9e2f31c0b9 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:55.592422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 642/1300] 10.222.0.33 () {56 vars in 1251 bytes} [Tue Jul 19 10:54:55 2022] GET /identity/v3/endpoints?service_id=41cb3534e17f4a17940479fe870be63c&interface=public®ion_id=RegionOne => generated 194 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:55.598044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:55.685555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:55.708120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923ff37-4e46-4478-a71f-865698294a44 None admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:55.743405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74892470-edc4-42ee-9af7-7f97094d945a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:55.744067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74892470-edc4-42ee-9af7-7f97094d945a None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:55.744352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74892470-edc4-42ee-9af7-7f97094d945a None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:55.744628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74892470-edc4-42ee-9af7-7f97094d945a None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:55.745340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74892470-edc4-42ee-9af7-7f97094d945a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:55.746699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74892470-edc4-42ee-9af7-7f97094d945a None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:55.754072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 659/1301] 10.222.0.33 () {56 vars in 1087 bytes} [Tue Jul 19 10:54:55 2022] GET /identity/v3/services => generated 2191 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 10:54:57.771468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c070214-926b-461f-82f7-d9fdd0e476e1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:57.772022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c070214-926b-461f-82f7-d9fdd0e476e1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:57.772022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c070214-926b-461f-82f7-d9fdd0e476e1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:57.773427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 643/1302] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:54:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:54:57.778767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:57.779192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:57.779318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:57.795775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:54:57.796999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:54:57.861471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 660/1303] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:54:57 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 19 10:54:57.884665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c070214-926b-461f-82f7-d9fdd0e476e1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:57.976908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:57.984090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c070214-926b-461f-82f7-d9fdd0e476e1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:58.016679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:58.017491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:58.017776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:58.018052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:58.018698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:58.020049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 19 10:54:58.031650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 19 10:54:58.032791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.get_or_create( Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application with Lock( Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self._enter() Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application return self.creator() Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application created_value = creator( Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: ERROR keystone.server.flask.application Jul 19 10:54:58.033637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 644/1304] 10.222.0.33 () {56 vars in 1117 bytes} [Tue Jul 19 10:54:57 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 19 10:54:58.038911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:58.117508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:54:58.140070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d360d7-5004-4677-8f09-2fa905f99596 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:58.172195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:58.172886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:58.173165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:58.173437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] PATH_INFO: `/v3/services` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:58.174127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] RBAC: Adding query filter params (name=placement) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 10:54:58.174427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:58.175777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b73af3-f37e-4d78-805c-13332f5406fe None admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:58.180731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 661/1305] 10.222.0.33 () {56 vars in 1116 bytes} [Tue Jul 19 10:54:58 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 10:54:58.187119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:54:58.283118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff464f6-8454-4f51-b700-8cf850ae21d3 None admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:54:58.314954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb8dd8d-c6b7-46d7-b58f-2c32f9838a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:54:58.315579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb8dd8d-c6b7-46d7-b58f-2c32f9838a3e None admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:54:58.315856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb8dd8d-c6b7-46d7-b58f-2c32f9838a3e None admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:54:58.316125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb8dd8d-c6b7-46d7-b58f-2c32f9838a3e None admin] PATH_INFO: `/v3/endpoints` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:54:58.316886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb8dd8d-c6b7-46d7-b58f-2c32f9838a3e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:54:58.318221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb8dd8d-c6b7-46d7-b58f-2c32f9838a3e None admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:54:58.335356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 645/1306] 10.222.0.33 () {60 vars in 1144 bytes} [Tue Jul 19 10:54:58 2022] POST /identity/v3/endpoints => generated 331 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 10:55:08.177763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=30 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:55:08.331234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:00.728463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485eb02-8149-4854-bde7-e38ca7b0c921 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:00.728463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485eb02-8149-4854-bde7-e38ca7b0c921 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:00.728463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485eb02-8149-4854-bde7-e38ca7b0c921 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:00.730006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 662/1307] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:00.744358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f5f250-fe76-44d5-8710-73b7f72189ad None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:00.747048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f5f250-fe76-44d5-8710-73b7f72189ad None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:00.748311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f5f250-fe76-44d5-8710-73b7f72189ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:00.846507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-08f5f250-fe76-44d5-8710-73b7f72189ad None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:00.847731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-08f5f250-fe76-44d5-8710-73b7f72189ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:00.980792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 646/1308] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:01.023895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba166a-546a-483e-9d6d-7685e7405144 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:01.024180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba166a-546a-483e-9d6d-7685e7405144 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:01.024946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba166a-546a-483e-9d6d-7685e7405144 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:01.030604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 663/1309] 10.222.0.33 () {54 vars in 965 bytes} [Tue Jul 19 10:56:01 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:01.036885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07797d04-427f-41d0-a579-d670b0fd3e43 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:01.037601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07797d04-427f-41d0-a579-d670b0fd3e43 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:01.037958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07797d04-427f-41d0-a579-d670b0fd3e43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:01.139807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-07797d04-427f-41d0-a579-d670b0fd3e43 None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:01.141133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-07797d04-427f-41d0-a579-d670b0fd3e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:01.162306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:01.223334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 647/1310] 10.222.0.33 () {58 vars in 1062 bytes} [Tue Jul 19 10:56:01 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:01.232307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ba166a-546a-483e-9d6d-7685e7405144 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:01.234520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-65ba166a-546a-483e-9d6d-7685e7405144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:01.351428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ba166a-546a-483e-9d6d-7685e7405144 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:01.432059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:01.432715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:01.433006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:01.433275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:01.434897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:01.491885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:01.548792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:01.550322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb5ee28-a455-4420-8a0e-783d4f265f47 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:01.569254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 664/1311] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:03.894612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7c0d5-deee-406a-a48c-bec8d8e0c524 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:03.895143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7c0d5-deee-406a-a48c-bec8d8e0c524 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:03.895143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7c0d5-deee-406a-a48c-bec8d8e0c524 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:03.896729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 648/1312] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:03.901753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c12ce-78b1-4e23-a125-b780c677837b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:03.902166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c12ce-78b1-4e23-a125-b780c677837b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:03.902390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c12ce-78b1-4e23-a125-b780c677837b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:03.995531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-0d8c12ce-78b1-4e23-a125-b780c677837b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:03.996972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0d8c12ce-78b1-4e23-a125-b780c677837b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:04.110551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 665/1313] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:04.151745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7c0d5-deee-406a-a48c-bec8d8e0c524 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:04.153325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bde7c0d5-deee-406a-a48c-bec8d8e0c524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:04.276263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7c0d5-deee-406a-a48c-bec8d8e0c524 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:04.353142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:04.354333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:04.355111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:04.355527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:04.355819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:04.358532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:04.468270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:04.469741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f278aaf-2481-4e06-b249-eb2c7a5b23ac service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:04.488532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 649/1314] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:06.823395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1596b483-82e0-4be3-b3c2-0e500ecbde0a None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:06.823930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1596b483-82e0-4be3-b3c2-0e500ecbde0a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:06.823930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1596b483-82e0-4be3-b3c2-0e500ecbde0a None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:06.825421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 666/1315] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:06.831324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6903cfd3-2e32-4736-b96c-1fe79c9fa7c9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:06.831760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6903cfd3-2e32-4736-b96c-1fe79c9fa7c9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:06.831887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6903cfd3-2e32-4736-b96c-1fe79c9fa7c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:06.895658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6903cfd3-2e32-4736-b96c-1fe79c9fa7c9 None None] MFA Rules not processed for user `89aa4c239d794594b3270fddfe326280`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:06.896890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6903cfd3-2e32-4736-b96c-1fe79c9fa7c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:06.948485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:06.993194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 650/1316] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:06 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:07.033145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1596b483-82e0-4be3-b3c2-0e500ecbde0a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:07.034682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1596b483-82e0-4be3-b3c2-0e500ecbde0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:07.044264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:07.136542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1596b483-82e0-4be3-b3c2-0e500ecbde0a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:07.207375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:07.208015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:07.208300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:07.208577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:07.210256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:07.273084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:07.319271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:07.320685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844e3df0-76b9-4f1e-a4dc-a48d32b530fc service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:07.338269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 667/1317] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:07 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:09.864042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b18497-ec52-4f9e-8352-1abdbe74cc0a None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:09.864575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b18497-ec52-4f9e-8352-1abdbe74cc0a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:09.864575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b18497-ec52-4f9e-8352-1abdbe74cc0a None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:09.865994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 651/1318] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:09.871331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb7156c-7821-4bea-bf04-3a7ebdf812fd None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:09.871737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb7156c-7821-4bea-bf04-3a7ebdf812fd None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:09.871879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb7156c-7821-4bea-bf04-3a7ebdf812fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:09.937279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cbb7156c-7821-4bea-bf04-3a7ebdf812fd None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:09.938494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cbb7156c-7821-4bea-bf04-3a7ebdf812fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:10.032056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 668/1319] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:09 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:10.070923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b18497-ec52-4f9e-8352-1abdbe74cc0a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:10.072424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-97b18497-ec52-4f9e-8352-1abdbe74cc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:10.173311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b18497-ec52-4f9e-8352-1abdbe74cc0a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:10.245338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:10.245970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:10.246250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:10.246521 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:10.248141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:10.255353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:10.353717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:10.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f25509-901d-4ee7-bf98-45ed5a180c38 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:10.373336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 652/1320] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:10 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:13.049299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa254848-70a2-46b7-9294-d9e4263faa4f None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:13.049847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa254848-70a2-46b7-9294-d9e4263faa4f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:13.049962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa254848-70a2-46b7-9294-d9e4263faa4f None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:13.051262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 669/1321] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:13.056646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedded16-c4ef-4853-bb22-e93ca5abb87e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:13.057065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedded16-c4ef-4853-bb22-e93ca5abb87e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:13.057231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedded16-c4ef-4853-bb22-e93ca5abb87e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:13.120834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-eedded16-c4ef-4853-bb22-e93ca5abb87e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:13.122023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-eedded16-c4ef-4853-bb22-e93ca5abb87e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:13.178850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:13.221173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 653/1322] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:13.257849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa254848-70a2-46b7-9294-d9e4263faa4f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:13.259411 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-aa254848-70a2-46b7-9294-d9e4263faa4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:13.279921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:13.369563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa254848-70a2-46b7-9294-d9e4263faa4f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:13.445733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:13.446376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:13.446658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:13.446928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:13.448550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:13.518387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:13.556879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:13.558297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6194663-f722-4240-8c40-60732d1fe2e9 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:13.578383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 670/1323] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 10:56:15.659400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd9c86-f3da-4275-b768-cf9e38a5a3c2 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:15.660133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd9c86-f3da-4275-b768-cf9e38a5a3c2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:15.660133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd9c86-f3da-4275-b768-cf9e38a5a3c2 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:15.661693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 654/1324] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:15.667185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffba5771-83f2-42b9-9758-04484f1fa583 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:15.667617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffba5771-83f2-42b9-9758-04484f1fa583 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:15.667896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffba5771-83f2-42b9-9758-04484f1fa583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:15.732185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ffba5771-83f2-42b9-9758-04484f1fa583 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:15.733387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ffba5771-83f2-42b9-9758-04484f1fa583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:15.829736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 671/1325] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:15.865369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd9c86-f3da-4275-b768-cf9e38a5a3c2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:15.867181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e4dd9c86-f3da-4275-b768-cf9e38a5a3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:15.970709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd9c86-f3da-4275-b768-cf9e38a5a3c2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:16.043411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:16.044063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:16.044342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:16.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:16.046809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:16.065317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:16.161597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:16.163026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf27b301-c370-445a-9b2d-581593d422d3 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:16.180802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 655/1326] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:18.277597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f563cb-ca6c-4a56-8b28-3add4b5db034 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:18.278102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f563cb-ca6c-4a56-8b28-3add4b5db034 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:18.278102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f563cb-ca6c-4a56-8b28-3add4b5db034 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:18.279495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 672/1327] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:18.284666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0126216-27b4-4875-a269-a4361c31bdc2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:18.285113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0126216-27b4-4875-a269-a4361c31bdc2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:18.285359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0126216-27b4-4875-a269-a4361c31bdc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:18.348762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b0126216-27b4-4875-a269-a4361c31bdc2 None None] MFA Rules not processed for user `89aa4c239d794594b3270fddfe326280`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:18.349964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b0126216-27b4-4875-a269-a4361c31bdc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:18.415797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:18.449975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 656/1328] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:18 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:18.488291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f563cb-ca6c-4a56-8b28-3add4b5db034 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:18.489877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b4f563cb-ca6c-4a56-8b28-3add4b5db034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:18.525171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:18.598097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f563cb-ca6c-4a56-8b28-3add4b5db034 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:18.669241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:18.669884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:18.670720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:18.670991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:18.672612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:18.753950 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:18.785954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:18.787387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098d5409-398f-4789-ae63-ebe64f3902fe service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:18.804910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 673/1329] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:18 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:21.093727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75562a9-d5ed-4da5-8df0-5dcb36ae113d None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:21.094248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75562a9-d5ed-4da5-8df0-5dcb36ae113d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:21.094248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75562a9-d5ed-4da5-8df0-5dcb36ae113d None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:21.095577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 657/1330] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:21.100886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6080d-03b0-4f5a-96c5-ac12017ea6d2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:21.101281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6080d-03b0-4f5a-96c5-ac12017ea6d2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:21.101454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6080d-03b0-4f5a-96c5-ac12017ea6d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:21.166061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bdd6080d-03b0-4f5a-96c5-ac12017ea6d2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:21.167266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bdd6080d-03b0-4f5a-96c5-ac12017ea6d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:21.260462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 674/1331] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:21.297213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75562a9-d5ed-4da5-8df0-5dcb36ae113d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:21.298726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f75562a9-d5ed-4da5-8df0-5dcb36ae113d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:21.398738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75562a9-d5ed-4da5-8df0-5dcb36ae113d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:21.471611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:21.472248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:21.472529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:21.472833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:21.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:21.501275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:21.582320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:21.583747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbea8637-d932-40e1-8320-e72375f679fe service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:21.602190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 658/1332] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:23.663148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73caa2ae-f53b-4a5e-994d-d7eb5ea6aef1 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:23.663683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73caa2ae-f53b-4a5e-994d-d7eb5ea6aef1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:23.663683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73caa2ae-f53b-4a5e-994d-d7eb5ea6aef1 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:23.665090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 675/1333] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:23.670818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62da7766-c02d-467c-9e7e-942878549e2f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:23.671198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62da7766-c02d-467c-9e7e-942878549e2f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:23.671372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62da7766-c02d-467c-9e7e-942878549e2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:23.735106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-62da7766-c02d-467c-9e7e-942878549e2f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:23.736316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-62da7766-c02d-467c-9e7e-942878549e2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:23.808610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:23.830902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 659/1334] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:23.877200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73caa2ae-f53b-4a5e-994d-d7eb5ea6aef1 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:23.878720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-73caa2ae-f53b-4a5e-994d-d7eb5ea6aef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:23.917742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:23.981211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73caa2ae-f53b-4a5e-994d-d7eb5ea6aef1 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:24.054087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:24.054722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:24.055005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:24.055278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:24.056928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:24.138217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:24.164968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:24.166403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde17c69-f4e6-420e-b9cf-9632824da7de service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:24.184013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 676/1335] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:26.539992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e09bea4-68d8-4197-a2be-765cf9484e09 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:26.540512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e09bea4-68d8-4197-a2be-765cf9484e09 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:26.540512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e09bea4-68d8-4197-a2be-765cf9484e09 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:26.541940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 660/1336] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:26.547050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c93bec1-28e2-4cd4-8fb2-58cf16cbda86 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:26.547431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c93bec1-28e2-4cd4-8fb2-58cf16cbda86 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:26.547614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c93bec1-28e2-4cd4-8fb2-58cf16cbda86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:26.611433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-2c93bec1-28e2-4cd4-8fb2-58cf16cbda86 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:26.612624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2c93bec1-28e2-4cd4-8fb2-58cf16cbda86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:26.708530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 677/1337] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:26 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:26.759908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f17af-f198-4e1c-a902-a786d0d82704 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:26.760193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f17af-f198-4e1c-a902-a786d0d82704 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:26.760465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f17af-f198-4e1c-a902-a786d0d82704 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:26.761872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 661/1338] 10.222.0.33 () {54 vars in 965 bytes} [Tue Jul 19 10:56:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:26.767618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac0d60-9eb9-45bc-b002-22d0cf9e80db None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:26.767913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac0d60-9eb9-45bc-b002-22d0cf9e80db None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:26.768198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac0d60-9eb9-45bc-b002-22d0cf9e80db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:26.819914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:26.840675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c1ac0d60-9eb9-45bc-b002-22d0cf9e80db None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:26.842380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c1ac0d60-9eb9-45bc-b002-22d0cf9e80db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:26.914853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 678/1339] 10.222.0.33 () {58 vars in 1062 bytes} [Tue Jul 19 10:56:26 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:26.924301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f17af-f198-4e1c-a902-a786d0d82704 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:26.925854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-885f17af-f198-4e1c-a902-a786d0d82704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:27.025228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f17af-f198-4e1c-a902-a786d0d82704 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:27.097490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:27.098126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:27.098407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:27.098673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:27.100286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:27.134014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:27.205746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:27.207172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1478aea-4437-4eb3-b1ba-2467d185f931 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:27.224506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 662/1340] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:26 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:31.734541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcce3704-e8f5-4e46-b61d-fe2092224d61 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:31.735091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcce3704-e8f5-4e46-b61d-fe2092224d61 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:31.735091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcce3704-e8f5-4e46-b61d-fe2092224d61 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:31.736507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 679/1341] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:31.742207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eb82cb-01a7-46e9-a50a-866e7e8f172d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:31.742654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eb82cb-01a7-46e9-a50a-866e7e8f172d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:31.742903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eb82cb-01a7-46e9-a50a-866e7e8f172d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:31.811179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f2eb82cb-01a7-46e9-a50a-866e7e8f172d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:31.812381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f2eb82cb-01a7-46e9-a50a-866e7e8f172d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:31.902807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:31.915325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 663/1342] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:31.955191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcce3704-e8f5-4e46-b61d-fe2092224d61 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:31.956749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fcce3704-e8f5-4e46-b61d-fe2092224d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:32.059906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcce3704-e8f5-4e46-b61d-fe2092224d61 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:32.098044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:32.135545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:32.136197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:32.136479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:32.136788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:32.138477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:32.250582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:32.252009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23997039-8773-444d-9811-2a0339eb5288 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:32.270099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 680/1343] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:34.943211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9c36f9-75b2-4d17-be69-524918c72971 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:34.943857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9c36f9-75b2-4d17-be69-524918c72971 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:34.943857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9c36f9-75b2-4d17-be69-524918c72971 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:34.945326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 664/1344] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:34.951159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29883b-cbf1-4ae9-ba73-09cf6c7e1475 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:34.951674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29883b-cbf1-4ae9-ba73-09cf6c7e1475 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:34.951780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29883b-cbf1-4ae9-ba73-09cf6c7e1475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:35.020896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1f29883b-cbf1-4ae9-ba73-09cf6c7e1475 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:35.022112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1f29883b-cbf1-4ae9-ba73-09cf6c7e1475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:35.040467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:35.118725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 681/1345] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:34 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:35.157127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9c36f9-75b2-4d17-be69-524918c72971 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:35.158725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6d9c36f9-75b2-4d17-be69-524918c72971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:35.260651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9c36f9-75b2-4d17-be69-524918c72971 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:35.332858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:35.333499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:35.333779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:35.334051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:35.335665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:35.377950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:35.444135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:35.445578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b660ce-164a-4bf1-ba83-82fb23a614e8 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:35.464387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 665/1346] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:35 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:39.426318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa68e48-9dca-4362-8222-0bb25add04b5 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:39.426872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa68e48-9dca-4362-8222-0bb25add04b5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:39.426872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa68e48-9dca-4362-8222-0bb25add04b5 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:39.428281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 682/1347] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:39.433606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273a43a-2c26-41b3-b446-d591f6e13d3e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:39.434046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273a43a-2c26-41b3-b446-d591f6e13d3e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:39.434216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273a43a-2c26-41b3-b446-d591f6e13d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:39.498964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b273a43a-2c26-41b3-b446-d591f6e13d3e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:39.500212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b273a43a-2c26-41b3-b446-d591f6e13d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:39.615171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:39.619869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 666/1348] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:39.658928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa68e48-9dca-4362-8222-0bb25add04b5 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:39.660868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9fa68e48-9dca-4362-8222-0bb25add04b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:39.772775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa68e48-9dca-4362-8222-0bb25add04b5 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:39.822607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:39.851114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:39.851762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:39.852043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:39.852315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:39.854003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:39.975363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:39.976828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81380197-a5fa-44d9-8092-06a56f7b52cc service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:39.995743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 683/1349] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:44.890187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a194c8-f4bf-404a-9759-198db7b033f5 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:44.890725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a194c8-f4bf-404a-9759-198db7b033f5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:44.890975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a194c8-f4bf-404a-9759-198db7b033f5 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:44.892457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 667/1350] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 10:56:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:44.898707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108d2cb6-3fb1-44eb-b9bb-a170bd642e9e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:44.899153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108d2cb6-3fb1-44eb-b9bb-a170bd642e9e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:44.899469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108d2cb6-3fb1-44eb-b9bb-a170bd642e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:44.988907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-108d2cb6-3fb1-44eb-b9bb-a170bd642e9e None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:44.990373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-108d2cb6-3fb1-44eb-b9bb-a170bd642e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:45.014176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:45.113498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 684/1351] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 10:56:44 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:49.083991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8aa955-42d1-47f4-bf2e-7a4e556b6419 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:49.084506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8aa955-42d1-47f4-bf2e-7a4e556b6419 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:49.084615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8aa955-42d1-47f4-bf2e-7a4e556b6419 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:49.086241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 668/1352] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 10:56:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:49.092366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bbb53e-9e4d-433d-8cd8-3937dc3df37b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:49.093202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bbb53e-9e4d-433d-8cd8-3937dc3df37b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:49.093660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bbb53e-9e4d-433d-8cd8-3937dc3df37b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:49.172350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-00bbb53e-9e4d-433d-8cd8-3937dc3df37b None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:49.173977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-00bbb53e-9e4d-433d-8cd8-3937dc3df37b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:49.252790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:49.295314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 685/1353] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 10:56:49 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:50.799209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f9b293-e334-4831-a571-0db9c241b6a6 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:50.799759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f9b293-e334-4831-a571-0db9c241b6a6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:50.799908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f9b293-e334-4831-a571-0db9c241b6a6 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:50.801338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 669/1354] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 10:56:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:50.806542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df894381-d2b6-4d34-a1d8-cf05343ee029 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:50.806957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df894381-d2b6-4d34-a1d8-cf05343ee029 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:50.807105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df894381-d2b6-4d34-a1d8-cf05343ee029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:50.883277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-df894381-d2b6-4d34-a1d8-cf05343ee029 None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:50.884472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-df894381-d2b6-4d34-a1d8-cf05343ee029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:50.992272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 686/1355] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 10:56:50 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:56:54.425479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d9628-610a-4e9d-9733-2961d7ccbae9 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:54.426483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d9628-610a-4e9d-9733-2961d7ccbae9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:54.426994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d9628-610a-4e9d-9733-2961d7ccbae9 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:54.428611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 670/1356] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:54.434263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d6358-5015-402a-8bbb-14a515fb0b1e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:54.435686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d6358-5015-402a-8bbb-14a515fb0b1e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:54.436169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d6358-5015-402a-8bbb-14a515fb0b1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:54.473064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:54.512099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ae6d6358-5015-402a-8bbb-14a515fb0b1e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:54.514061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ae6d6358-5015-402a-8bbb-14a515fb0b1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:54.618337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 687/1357] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:56:54.685924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89e2a24-6154-4fee-82fa-1bd5681bead3 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:54.686344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89e2a24-6154-4fee-82fa-1bd5681bead3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:54.686770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89e2a24-6154-4fee-82fa-1bd5681bead3 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:54.689131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 671/1358] 10.222.0.33 () {54 vars in 962 bytes} [Tue Jul 19 10:56:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:54.696621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8973f2e4-f864-4796-9c8c-8c1ee78dd06d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:54.696935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8973f2e4-f864-4796-9c8c-8c1ee78dd06d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:54.697208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8973f2e4-f864-4796-9c8c-8c1ee78dd06d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:54.777711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8973f2e4-f864-4796-9c8c-8c1ee78dd06d None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:54.778938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8973f2e4-f864-4796-9c8c-8c1ee78dd06d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:54.833776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:54.905411 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 688/1359] 10.222.0.33 () {58 vars in 1059 bytes} [Tue Jul 19 10:56:54 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:56:54.913098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89e2a24-6154-4fee-82fa-1bd5681bead3 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:54.914705 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a89e2a24-6154-4fee-82fa-1bd5681bead3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:55.075804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89e2a24-6154-4fee-82fa-1bd5681bead3 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:55.151938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:55.195776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:55.196430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:55.196733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:55.197013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:55.198636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:55.296315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:55.297949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a16d8-d08b-4923-b96a-d4ec09d58858 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:55.321169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 672/1360] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:56:57.593637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4af2f6-e963-4391-8d63-800a5f0dd022 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:57.594104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4af2f6-e963-4391-8d63-800a5f0dd022 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:57.594206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4af2f6-e963-4391-8d63-800a5f0dd022 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:57.599563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 689/1361] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:56:57 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:56:57.600966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d472fd-333d-4ca3-9253-a10885769aec None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:57.601426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d472fd-333d-4ca3-9253-a10885769aec None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:57.601522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d472fd-333d-4ca3-9253-a10885769aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:57.685687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-70d472fd-333d-4ca3-9253-a10885769aec None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:56:57.687226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-70d472fd-333d-4ca3-9253-a10885769aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:57.728539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:57.812778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 673/1362] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:56:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:56:57.856782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4af2f6-e963-4391-8d63-800a5f0dd022 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:56:57.858536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1b4af2f6-e963-4391-8d63-800a5f0dd022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:58.032964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4af2f6-e963-4391-8d63-800a5f0dd022 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:56:58.043136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:56:58.155532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:56:58.156178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:56:58.156461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:56:58.156760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:56:58.158410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:56:58.250437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:56:58.252007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e2cbbc-32d5-4347-bfd6-b21fbd092aa1 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:56:58.270388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 690/1363] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:56:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:00.665897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54043061-cd87-43ee-917f-066bd54c8752 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:00.666480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54043061-cd87-43ee-917f-066bd54c8752 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:00.666798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54043061-cd87-43ee-917f-066bd54c8752 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:00.668161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 674/1364] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:00.674105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a157eb1-186b-416d-8e69-9f9f2d4fb597 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:00.674546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a157eb1-186b-416d-8e69-9f9f2d4fb597 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:00.674656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a157eb1-186b-416d-8e69-9f9f2d4fb597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:00.689502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:00.748779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7a157eb1-186b-416d-8e69-9f9f2d4fb597 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:00.749983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7a157eb1-186b-416d-8e69-9f9f2d4fb597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:00.855153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 691/1365] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:00.900316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8347f68-a784-4a3f-891f-fb637134005d None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:00.900606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8347f68-a784-4a3f-891f-fb637134005d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:00.900913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8347f68-a784-4a3f-891f-fb637134005d None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:00.902409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 675/1366] 10.222.0.33 () {54 vars in 962 bytes} [Tue Jul 19 10:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:00.908751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dd1ad6-cc2a-42c8-aecc-74ab17b7a6a0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:00.909174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dd1ad6-cc2a-42c8-aecc-74ab17b7a6a0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:00.909459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dd1ad6-cc2a-42c8-aecc-74ab17b7a6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:00.990974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d3dd1ad6-cc2a-42c8-aecc-74ab17b7a6a0 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:00.992483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d3dd1ad6-cc2a-42c8-aecc-74ab17b7a6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:01.030015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:01.127912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 692/1367] 10.222.0.33 () {58 vars in 1059 bytes} [Tue Jul 19 10:57:00 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:01.135422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8347f68-a784-4a3f-891f-fb637134005d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:01.136993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b8347f68-a784-4a3f-891f-fb637134005d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:01.303450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8347f68-a784-4a3f-891f-fb637134005d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:01.326547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:01.425184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:01.425954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:01.426240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:01.426509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:01.428165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:01.520448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:01.522389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caead273-c5c9-4358-a060-d8642cf5bce0 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:01.543734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 676/1368] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:02.513294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28936c1a-7ee2-4b69-a739-b0d994760917 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:02.513838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28936c1a-7ee2-4b69-a739-b0d994760917 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:02.513838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28936c1a-7ee2-4b69-a739-b0d994760917 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:02.515398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 693/1369] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:02.520631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de27729c-4fd9-4eaa-b774-3a5e349d9d50 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:02.521152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de27729c-4fd9-4eaa-b774-3a5e349d9d50 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:02.521265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de27729c-4fd9-4eaa-b774-3a5e349d9d50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:02.553819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:02.591953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-de27729c-4fd9-4eaa-b774-3a5e349d9d50 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:02.593195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-de27729c-4fd9-4eaa-b774-3a5e349d9d50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:02.695354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 677/1370] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:03.044737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9760d-3599-466a-adc4-f8c6224d6afc None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:03.044737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9760d-3599-466a-adc4-f8c6224d6afc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:03.044737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9760d-3599-466a-adc4-f8c6224d6afc None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:03.046703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 694/1371] 10.222.0.33 () {54 vars in 964 bytes} [Tue Jul 19 10:57:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:03.051612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf910da-e2f5-46b0-ab3b-0ec1fd1af1ff None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:03.051977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf910da-e2f5-46b0-ab3b-0ec1fd1af1ff None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:03.052327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf910da-e2f5-46b0-ab3b-0ec1fd1af1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:03.132469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6cf910da-e2f5-46b0-ab3b-0ec1fd1af1ff None None] MFA Rules not processed for user `102f9668a5f948f99ac4ba6fc639c8ad`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:03.133809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6cf910da-e2f5-46b0-ab3b-0ec1fd1af1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:03.181932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:03.214898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 678/1372] 10.222.0.33 () {58 vars in 1061 bytes} [Tue Jul 19 10:57:03 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:03.222813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e9760d-3599-466a-adc4-f8c6224d6afc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:03.224330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-96e9760d-3599-466a-adc4-f8c6224d6afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:03.339012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e9760d-3599-466a-adc4-f8c6224d6afc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:03.383767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:03.420260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:03.420935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:03.421219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:03.421490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:03.423425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:03.537548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:03.539142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c710c64-348f-4da1-b782-d014d9bf779e service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:03.560271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 695/1373] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:03.678159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0438259-8805-4942-b082-a15bcbffe2c9 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:03.678589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0438259-8805-4942-b082-a15bcbffe2c9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:03.679051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0438259-8805-4942-b082-a15bcbffe2c9 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:03.681370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 679/1374] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:03.686993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95af0eb4-a545-4041-a2fd-63fc3e2b3606 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:03.687283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95af0eb4-a545-4041-a2fd-63fc3e2b3606 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:03.687555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95af0eb4-a545-4041-a2fd-63fc3e2b3606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:03.760945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-95af0eb4-a545-4041-a2fd-63fc3e2b3606 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:03.762418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-95af0eb4-a545-4041-a2fd-63fc3e2b3606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:03.785916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:03.875332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 696/1375] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:57:03.921028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0438259-8805-4942-b082-a15bcbffe2c9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:03.922988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a0438259-8805-4942-b082-a15bcbffe2c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:04.084194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0438259-8805-4942-b082-a15bcbffe2c9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:04.136589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:04.198588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:04.199236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:04.199520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:04.199793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:04.201473 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:04.297923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:04.299519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cbada9-4a1a-43e9-98a3-b5387064a2dc service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:04.317645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 680/1376] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:06.379900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74431795-93fb-4433-b431-1a93bcc70df4 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:06.380377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74431795-93fb-4433-b431-1a93bcc70df4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:06.380458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74431795-93fb-4433-b431-1a93bcc70df4 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:06.381872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 697/1377] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:06.387167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf359f1-8c81-4c5f-9a34-0a0942cca184 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:06.387608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf359f1-8c81-4c5f-9a34-0a0942cca184 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:06.387879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf359f1-8c81-4c5f-9a34-0a0942cca184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:06.459323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-adf359f1-8c81-4c5f-9a34-0a0942cca184 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:06.460532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-adf359f1-8c81-4c5f-9a34-0a0942cca184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:06.467855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:06.575991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 681/1378] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:57:06.621135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74431795-93fb-4433-b431-1a93bcc70df4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:06.622728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-74431795-93fb-4433-b431-1a93bcc70df4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:06.770602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74431795-93fb-4433-b431-1a93bcc70df4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:06.774374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:06.899341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:06.901125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:06.901125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:06.901125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:06.902313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:07.027887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:07.029574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed5968c-f4c4-4821-971f-b81496da7ded service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:07.060881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 698/1379] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:09.146673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c0547-98d4-49be-a88a-3bae464e47e5 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:09.147188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c0547-98d4-49be-a88a-3bae464e47e5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:09.147188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c0547-98d4-49be-a88a-3bae464e47e5 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:09.148509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 682/1380] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:09.153971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae66f73-5753-466d-a62f-53fba60d2b38 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:09.154373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae66f73-5753-466d-a62f-53fba60d2b38 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:09.154699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae66f73-5753-466d-a62f-53fba60d2b38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:09.165069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:09.222662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8ae66f73-5753-466d-a62f-53fba60d2b38 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:09.224000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8ae66f73-5753-466d-a62f-53fba60d2b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:09.326308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 699/1381] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:09.378284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c0547-98d4-49be-a88a-3bae464e47e5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:09.379886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-235c0547-98d4-49be-a88a-3bae464e47e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:09.525567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:09.537717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c0547-98d4-49be-a88a-3bae464e47e5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:09.655831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:09.656591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:09.656904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:09.657176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:09.658836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:09.756271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:09.757809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6fdd10-5289-4bf6-a6ae-29e161b0ca5a service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:09.769798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:09.779459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 683/1382] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:12.042715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c82045-2881-4d30-922c-eff9977bf76c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:12.043264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c82045-2881-4d30-922c-eff9977bf76c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:12.043494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c82045-2881-4d30-922c-eff9977bf76c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:12.045319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 700/1383] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:12.051314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11277f-cb8c-4b72-b460-03f88ae163a3 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:12.051876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11277f-cb8c-4b72-b460-03f88ae163a3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:12.051969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11277f-cb8c-4b72-b460-03f88ae163a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:12.119203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-eb11277f-cb8c-4b72-b460-03f88ae163a3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:12.120458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-eb11277f-cb8c-4b72-b460-03f88ae163a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:12.221022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 684/1384] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:12.267133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c82045-2881-4d30-922c-eff9977bf76c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:12.268666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-32c82045-2881-4d30-922c-eff9977bf76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:12.356439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:12.436209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c82045-2881-4d30-922c-eff9977bf76c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:12.568369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:12.569141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:12.569445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:12.569852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:12.571793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:12.627579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:12.671939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:12.673657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d34308-fc56-44ab-ad27-c104362f7dce service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:12.692765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 701/1385] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:14.756918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1965fd6-dfc9-4450-8206-e1e14b61a743 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:14.757460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1965fd6-dfc9-4450-8206-e1e14b61a743 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:14.757460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1965fd6-dfc9-4450-8206-e1e14b61a743 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:14.758899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 685/1386] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:57:14.764882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257a384d-9b73-42ac-a910-fbc1090a5614 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:14.765360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257a384d-9b73-42ac-a910-fbc1090a5614 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:14.765486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257a384d-9b73-42ac-a910-fbc1090a5614 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:14.832868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-257a384d-9b73-42ac-a910-fbc1090a5614 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:14.834285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-257a384d-9b73-42ac-a910-fbc1090a5614 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:14.940038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 702/1387] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:14.985997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1965fd6-dfc9-4450-8206-e1e14b61a743 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:14.987510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e1965fd6-dfc9-4450-8206-e1e14b61a743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:15.069454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:15.140756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1965fd6-dfc9-4450-8206-e1e14b61a743 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:15.253739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:15.254393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:15.254676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:15.254962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:15.256650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:15.295128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:15.347531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:15.349448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de4a1c7-c9f6-40a2-850b-1599fe4c2b92 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:15.370359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 686/1388] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:17.522811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8722be76-e258-4655-8983-94765d7b7b41 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:17.523355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8722be76-e258-4655-8983-94765d7b7b41 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:17.523458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8722be76-e258-4655-8983-94765d7b7b41 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:17.524957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 703/1389] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:17.531087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e733068c-d07b-4770-a860-a2efa63df4c2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:17.531512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e733068c-d07b-4770-a860-a2efa63df4c2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:17.531708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e733068c-d07b-4770-a860-a2efa63df4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:17.601607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e733068c-d07b-4770-a860-a2efa63df4c2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:17.603009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e733068c-d07b-4770-a860-a2efa63df4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:17.707536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 687/1390] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:17.752071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8722be76-e258-4655-8983-94765d7b7b41 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:17.753630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8722be76-e258-4655-8983-94765d7b7b41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:17.784764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:17.905539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8722be76-e258-4655-8983-94765d7b7b41 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:18.006841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:18.018640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:18.019289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:18.019576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:18.019865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:18.021566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:18.127638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:18.129199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9637e979-b655-42b4-b7cd-a56a86dd0096 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:18.147630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 704/1391] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:20.346217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f712e6af-7e40-4929-ba96-77ccf63dc390 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:20.346758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f712e6af-7e40-4929-ba96-77ccf63dc390 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:20.346877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f712e6af-7e40-4929-ba96-77ccf63dc390 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:20.348340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 688/1392] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:20.354147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502ebd4-75ed-41d0-92ec-5942160cdbff None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:20.354582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502ebd4-75ed-41d0-92ec-5942160cdbff None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:20.354710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502ebd4-75ed-41d0-92ec-5942160cdbff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:20.420227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7502ebd4-75ed-41d0-92ec-5942160cdbff None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:20.421187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7502ebd4-75ed-41d0-92ec-5942160cdbff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:20.478558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:20.517910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 705/1393] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:20.562625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f712e6af-7e40-4929-ba96-77ccf63dc390 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:20.564188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f712e6af-7e40-4929-ba96-77ccf63dc390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:20.580750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:20.734425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f712e6af-7e40-4929-ba96-77ccf63dc390 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:20.829187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:20.856303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:20.856989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:20.857271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:20.857541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:20.859199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:20.948871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:20.951083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37774250-5df0-4874-9d78-6c177357b25f service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:20.969786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 689/1394] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:23.114269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820a8fd-c4c3-4eda-a19e-974a884b33ff None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:23.114800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820a8fd-c4c3-4eda-a19e-974a884b33ff None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:23.114911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820a8fd-c4c3-4eda-a19e-974a884b33ff None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:23.116576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 706/1395] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:23.123147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e3b5c7-4e6a-4c06-b5e6-783121664d92 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:23.123669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e3b5c7-4e6a-4c06-b5e6-783121664d92 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:23.123762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e3b5c7-4e6a-4c06-b5e6-783121664d92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:23.197905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e8e3b5c7-4e6a-4c06-b5e6-783121664d92 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:23.199080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e8e3b5c7-4e6a-4c06-b5e6-783121664d92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:23.262073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:23.319207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 690/1396] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:23.370476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820a8fd-c4c3-4eda-a19e-974a884b33ff None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:23.372666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d820a8fd-c4c3-4eda-a19e-974a884b33ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:23.525492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820a8fd-c4c3-4eda-a19e-974a884b33ff None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:23.586819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:23.666796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:23.667730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:23.668297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:23.669376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:23.671769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:23.789242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:23.790766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f7cef9-acad-44b8-aad5-690a66e1ea7d service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:23.814401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 707/1397] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:25.892611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07643b8d-2253-4a0a-ac03-7be4d688881c None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:25.893307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07643b8d-2253-4a0a-ac03-7be4d688881c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:25.893307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07643b8d-2253-4a0a-ac03-7be4d688881c None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:25.894582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 691/1398] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:25.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da561949-31cd-498b-99ef-c60a9006ec04 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:25.900479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da561949-31cd-498b-99ef-c60a9006ec04 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:25.900574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da561949-31cd-498b-99ef-c60a9006ec04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:25.920541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307ae0e-1632-400a-960f-ad430ee0eb98 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:25.921075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307ae0e-1632-400a-960f-ad430ee0eb98 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:25.921355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307ae0e-1632-400a-960f-ad430ee0eb98 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:25.923140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 692/1399] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:57:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 10:57:25.928392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da5603a-0977-4980-bc12-e20e9d78b59a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:25.928676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da5603a-0977-4980-bc12-e20e9d78b59a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:25.928987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da5603a-0977-4980-bc12-e20e9d78b59a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:25.968005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-da561949-31cd-498b-99ef-c60a9006ec04 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:25.969239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-da561949-31cd-498b-99ef-c60a9006ec04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:25.971711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:25.998652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7da5603a-0977-4980-bc12-e20e9d78b59a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:25.999870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7da5603a-0977-4980-bc12-e20e9d78b59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:26.070308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 708/1400] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:26.098918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 693/1401] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:57:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:57:26.126155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f25c4-590d-4d25-a278-63d2493a39a1 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:26.126512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f25c4-590d-4d25-a278-63d2493a39a1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:26.126855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f25c4-590d-4d25-a278-63d2493a39a1 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:26.128324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 709/1402] 10.222.0.33 () {54 vars in 961 bytes} [Tue Jul 19 10:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:26.133517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:26.133812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:26.134097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:26.145820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324f25c4-590d-4d25-a278-63d2493a39a1 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:26.147305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-324f25c4-590d-4d25-a278-63d2493a39a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:26.170156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:26.211418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] MFA Rules not processed for user `9f86b27721074978b12f34dd591362ce`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:26.212638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:26.276365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:26.288484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 694/1403] 10.222.0.33 () {58 vars in 1058 bytes} [Tue Jul 19 10:57:26 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:26.296016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:26.297980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324f25c4-590d-4d25-a278-63d2493a39a1 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:26.303387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee7ee4-4b64-4a6e-a4af-6ca779b1ee0a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:26.384470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:26.385146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:26.385423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:26.385691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:26.387380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:26.416990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:26.417563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:26.417847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:26.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:26.420270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:26.435019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:26.498571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:26.500024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d585d0e-f10e-4e04-8ada-b2a104f3dea1 service glance] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:26.512559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:26.514166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cc4dd9-a495-4bdd-959a-9c57cea97f11 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:26.519002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:26.519275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 695/1404] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 10:57:26.533513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 710/1405] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 10:57:27.230221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4958c01b-6779-42b6-b175-06be167c109f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:27.230698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4958c01b-6779-42b6-b175-06be167c109f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:27.230884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4958c01b-6779-42b6-b175-06be167c109f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:27.297186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4958c01b-6779-42b6-b175-06be167c109f None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:27.298414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4958c01b-6779-42b6-b175-06be167c109f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:27.384689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 696/1406] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 10:57:27 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:27.430562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393e0aaa-f70b-435e-8bd2-51f7bfdda3d6 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:27.430998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393e0aaa-f70b-435e-8bd2-51f7bfdda3d6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:27.431161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393e0aaa-f70b-435e-8bd2-51f7bfdda3d6 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:27.432577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 711/1407] 10.222.0.33 () {54 vars in 943 bytes} [Tue Jul 19 10:57:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:27.439244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909061a1-94c0-4113-88e1-932999c71f09 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:27.439535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909061a1-94c0-4113-88e1-932999c71f09 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:27.439813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909061a1-94c0-4113-88e1-932999c71f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:27.455163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:27.516388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-909061a1-94c0-4113-88e1-932999c71f09 None None] MFA Rules not processed for user `110d74df72174fba99493dedd2e98d08`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:27.517654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-909061a1-94c0-4113-88e1-932999c71f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:27.628073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 697/1408] 10.222.0.33 () {58 vars in 1040 bytes} [Tue Jul 19 10:57:27 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:27.637983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393e0aaa-f70b-435e-8bd2-51f7bfdda3d6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:27.639465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-393e0aaa-f70b-435e-8bd2-51f7bfdda3d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:27.773549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393e0aaa-f70b-435e-8bd2-51f7bfdda3d6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:27.850193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:27.878612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:27.879257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:27.879538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:27.879812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:27.881521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:27.962398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:27.963983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bafda1c-973f-4add-aef6-492636108cc7 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:27.971997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 712/1409] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 10:57:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 10:57:37.626315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=68 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:37.968848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=49 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:50.028448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc639e47-befd-4edb-9452-4773b904ff77 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:50.029039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc639e47-befd-4edb-9452-4773b904ff77 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:50.029039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc639e47-befd-4edb-9452-4773b904ff77 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:50.030453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 698/1410] 10.222.0.33 () {54 vars in 918 bytes} [Tue Jul 19 10:57:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:50.035816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b974208-505c-474a-9643-d9bf6dc1caca None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:50.036207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b974208-505c-474a-9643-d9bf6dc1caca None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:50.036407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b974208-505c-474a-9643-d9bf6dc1caca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:50.109967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7b974208-505c-474a-9643-d9bf6dc1caca None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:50.111169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7b974208-505c-474a-9643-d9bf6dc1caca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:50.213879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 713/1411] 10.222.0.33 () {58 vars in 1015 bytes} [Tue Jul 19 10:57:50 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:50.229273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d4c0c8-f964-4697-84ab-8ffc89ee6ec7 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:50.229561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d4c0c8-f964-4697-84ab-8ffc89ee6ec7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:50.229833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d4c0c8-f964-4697-84ab-8ffc89ee6ec7 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:50.231203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 699/1412] 10.222.0.33 () {54 vars in 918 bytes} [Tue Jul 19 10:57:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:50.235792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed304ee-2f5b-4801-8ae0-2ae01fd6eb19 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:50.236076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed304ee-2f5b-4801-8ae0-2ae01fd6eb19 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:50.236348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed304ee-2f5b-4801-8ae0-2ae01fd6eb19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:50.306591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:50.309884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-3ed304ee-2f5b-4801-8ae0-2ae01fd6eb19 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:50.311646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3ed304ee-2f5b-4801-8ae0-2ae01fd6eb19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:50.425028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 714/1413] 10.222.0.33 () {58 vars in 1015 bytes} [Tue Jul 19 10:57:50 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:52.897935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4861f8c-0393-4196-a276-63d13895f94b None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:52.898456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4861f8c-0393-4196-a276-63d13895f94b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:52.898456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4861f8c-0393-4196-a276-63d13895f94b None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:52.899776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 700/1414] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:57:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:52.905705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd5238d-acec-4487-9095-11d0c4567090 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:52.906121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd5238d-acec-4487-9095-11d0c4567090 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:52.906264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd5238d-acec-4487-9095-11d0c4567090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:52.972209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cdd5238d-acec-4487-9095-11d0c4567090 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:52.973450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cdd5238d-acec-4487-9095-11d0c4567090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:53.035373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:53.070570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 715/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:57:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:53.114958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4861f8c-0393-4196-a276-63d13895f94b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:53.116896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d4861f8c-0393-4196-a276-63d13895f94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:53.260745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4861f8c-0393-4196-a276-63d13895f94b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:53.330537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:53.369765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e791fb15-4319-4356-a177-a1603095959c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:53.370401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e791fb15-4319-4356-a177-a1603095959c service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:53.370680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e791fb15-4319-4356-a177-a1603095959c service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:53.370950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e791fb15-4319-4356-a177-a1603095959c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:53.373173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e791fb15-4319-4356-a177-a1603095959c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:53.460010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e791fb15-4319-4356-a177-a1603095959c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:53.461606 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e791fb15-4319-4356-a177-a1603095959c service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:53.479553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 701/1416] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:54.368781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab02184b-c75f-4346-a02c-80f735775f04 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:54.369307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab02184b-c75f-4346-a02c-80f735775f04 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:54.369307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab02184b-c75f-4346-a02c-80f735775f04 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:54.370669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 716/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:57:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:54.375798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e72012-0403-4b04-b210-970bc69cd3c5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:54.376220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e72012-0403-4b04-b210-970bc69cd3c5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:54.376349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e72012-0403-4b04-b210-970bc69cd3c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:54.450601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-83e72012-0403-4b04-b210-970bc69cd3c5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:54.451796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-83e72012-0403-4b04-b210-970bc69cd3c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:54.477352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:54.545564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 702/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:57:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:54.696823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab02184b-c75f-4346-a02c-80f735775f04 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:54.698528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ab02184b-c75f-4346-a02c-80f735775f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:54.844750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab02184b-c75f-4346-a02c-80f735775f04 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:54.885440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:54.958334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:54.958923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:54.959205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:54.959481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:54.961148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:55.048307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:55.049914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf93ce8-4a41-4c5e-a03a-6c1c3b1d92be service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:55.067977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 717/1419] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:55.947905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259af56-d8e5-4c7d-a35e-83e9c275ee38 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:55.948487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259af56-d8e5-4c7d-a35e-83e9c275ee38 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:55.948487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259af56-d8e5-4c7d-a35e-83e9c275ee38 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:55.949954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 703/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:55.956245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c2baa9-6e60-44cd-ae98-7fb55a988585 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:55.956688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c2baa9-6e60-44cd-ae98-7fb55a988585 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:55.956835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c2baa9-6e60-44cd-ae98-7fb55a988585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:56.008852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:56.030720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-41c2baa9-6e60-44cd-ae98-7fb55a988585 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:56.031918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-41c2baa9-6e60-44cd-ae98-7fb55a988585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:56.128363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 718/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:57:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:56.289908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259af56-d8e5-4c7d-a35e-83e9c275ee38 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:56.291337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c259af56-d8e5-4c7d-a35e-83e9c275ee38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:56.440206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259af56-d8e5-4c7d-a35e-83e9c275ee38 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:56.451992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:56.550909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:56.551483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:56.551764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:56.552038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:56.553706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:56.646603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:56.648200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c3a0e-56f3-4720-b2ae-db1092e7e7a1 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:56.666196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 704/1422] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:57.522887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a127d1d1-74d5-44dc-b0b1-4bfffd12022b None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:57.523453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a127d1d1-74d5-44dc-b0b1-4bfffd12022b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:57.523564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a127d1d1-74d5-44dc-b0b1-4bfffd12022b None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:57.525003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 719/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:57:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:57.530337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540bfa0-26f2-4a50-a6e2-57b7fdcef32f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:57.530780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540bfa0-26f2-4a50-a6e2-57b7fdcef32f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:57.530885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540bfa0-26f2-4a50-a6e2-57b7fdcef32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:57.549601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:57.604461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b540bfa0-26f2-4a50-a6e2-57b7fdcef32f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:57.606297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b540bfa0-26f2-4a50-a6e2-57b7fdcef32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:57.701186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 705/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:57:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:57.850747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a127d1d1-74d5-44dc-b0b1-4bfffd12022b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:57.852488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a127d1d1-74d5-44dc-b0b1-4bfffd12022b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:57.976303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:58.012513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a127d1d1-74d5-44dc-b0b1-4bfffd12022b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:58.122633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:58.123206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:58.123485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:58.123758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:58.125417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:58.199628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:58.212905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:58.214482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10410d3c-4b1f-41d5-b350-68e89c330c3f service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:58.235568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 720/1425] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:57:59.090841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0efe7cb-9bcd-49d3-b754-8d885426b914 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:59.091400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0efe7cb-9bcd-49d3-b754-8d885426b914 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:59.091400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0efe7cb-9bcd-49d3-b754-8d885426b914 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:59.092779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 706/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:57:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:57:59.097683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1971fd1-de34-4b5b-9265-d07e27c65f9a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:59.098121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1971fd1-de34-4b5b-9265-d07e27c65f9a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:59.098224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1971fd1-de34-4b5b-9265-d07e27c65f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:59.171880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b1971fd1-de34-4b5b-9265-d07e27c65f9a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:57:59.173123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b1971fd1-de34-4b5b-9265-d07e27c65f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:59.267959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 721/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:57:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:57:59.416590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0efe7cb-9bcd-49d3-b754-8d885426b914 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:57:59.418131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b0efe7cb-9bcd-49d3-b754-8d885426b914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:59.500478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:59.563403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0efe7cb-9bcd-49d3-b754-8d885426b914 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:57:59.673018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:57:59.673657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:57:59.673942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:57:59.674218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:57:59.675863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:57:59.723323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:57:59.764102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:57:59.765731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1f0aa4-d446-4a1d-8e85-31d041a8aebd service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:57:59.783838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 707/1428] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:57:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:00.582484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7094d00c-0ab2-4068-947d-1c3825f97cfb None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:00.583036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7094d00c-0ab2-4068-947d-1c3825f97cfb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:00.583036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7094d00c-0ab2-4068-947d-1c3825f97cfb None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:00.584405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 722/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:58:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:00.589495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5490ab62-d343-491f-a340-7eff157e30a6 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:00.589912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5490ab62-d343-491f-a340-7eff157e30a6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:00.590057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5490ab62-d343-491f-a340-7eff157e30a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:00.671121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5490ab62-d343-491f-a340-7eff157e30a6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:00.672333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5490ab62-d343-491f-a340-7eff157e30a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:00.769020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 708/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:58:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:00.918396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7094d00c-0ab2-4068-947d-1c3825f97cfb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:00.919936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7094d00c-0ab2-4068-947d-1c3825f97cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:00.968405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:01.064573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7094d00c-0ab2-4068-947d-1c3825f97cfb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:01.175306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:01.175954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:01.176240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:01.176515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:01.178253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:01.185748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:01.267858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:01.269436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4cc8ac-3589-4672-82b6-60ba6ab0a8db service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:01.287485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 723/1431] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:02.089191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaa64f0-affe-4739-bfcb-5f61020f255d None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:02.089755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaa64f0-affe-4739-bfcb-5f61020f255d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:02.089755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaa64f0-affe-4739-bfcb-5f61020f255d None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:02.091136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 709/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:02.096162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35577505-40be-4b6d-a5d7-6605c504e45e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:02.096581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35577505-40be-4b6d-a5d7-6605c504e45e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:02.096742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35577505-40be-4b6d-a5d7-6605c504e45e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:02.169738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-35577505-40be-4b6d-a5d7-6605c504e45e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:02.170935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-35577505-40be-4b6d-a5d7-6605c504e45e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:02.237389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:02.268371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 724/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:58:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:02.415892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaa64f0-affe-4739-bfcb-5f61020f255d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:02.417436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6eaa64f0-affe-4739-bfcb-5f61020f255d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:02.435718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:02.566118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaa64f0-affe-4739-bfcb-5f61020f255d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:02.651499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:02.678649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:02.679293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:02.679579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:02.679856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:02.681552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:02.771642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:02.773278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4a6817-ec0b-4a89-95f4-f55adb098dd3 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:02.790769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 710/1434] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:03.594909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b4e79-c467-40c7-a7a2-7c3d2c1a1227 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:03.595472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b4e79-c467-40c7-a7a2-7c3d2c1a1227 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:03.595472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b4e79-c467-40c7-a7a2-7c3d2c1a1227 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:03.596903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 725/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:58:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:03.602266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d321d1-84dc-4658-bd29-7147a98bae62 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:03.602681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d321d1-84dc-4658-bd29-7147a98bae62 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:03.602849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d321d1-84dc-4658-bd29-7147a98bae62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:03.677042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a9d321d1-84dc-4658-bd29-7147a98bae62 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:03.678251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a9d321d1-84dc-4658-bd29-7147a98bae62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:03.718486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:03.772737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 711/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:58:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:03.931226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65b4e79-c467-40c7-a7a2-7c3d2c1a1227 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:03.932854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b65b4e79-c467-40c7-a7a2-7c3d2c1a1227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:04.085949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65b4e79-c467-40c7-a7a2-7c3d2c1a1227 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:04.138462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:04.196540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:04.197144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:04.197424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:04.197691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:04.199901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:04.290602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:04.292208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165c68a0-aae4-494d-bf00-8cfe2638b85b service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:04.311338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 726/1437] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:05.121816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc43b586-5304-44fd-a00b-f64056d71cc5 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:05.122327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc43b586-5304-44fd-a00b-f64056d71cc5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:05.122327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc43b586-5304-44fd-a00b-f64056d71cc5 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:05.123675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 712/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 10:58:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:05.128625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9efc42c-afce-4ab2-a86e-40236d4b17a8 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:05.129061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9efc42c-afce-4ab2-a86e-40236d4b17a8 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:05.129344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9efc42c-afce-4ab2-a86e-40236d4b17a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:05.202827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c9efc42c-afce-4ab2-a86e-40236d4b17a8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:05.204024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c9efc42c-afce-4ab2-a86e-40236d4b17a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:05.206037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:05.300243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 727/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 10:58:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:05.450003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc43b586-5304-44fd-a00b-f64056d71cc5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:05.451543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cc43b586-5304-44fd-a00b-f64056d71cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:05.595102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc43b586-5304-44fd-a00b-f64056d71cc5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:05.623068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:05.709080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:05.709655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:05.709934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:05.710207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:05.711849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:05.799987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:05.801597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c675e6-00a5-4011-bc3c-3153502f8177 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:05.819185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 713/1440] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:07.734438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e691525e-da8a-492d-8162-9ae36ad34bde None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:07.735010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e691525e-da8a-492d-8162-9ae36ad34bde None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:07.735010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e691525e-da8a-492d-8162-9ae36ad34bde None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:07.736372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 728/1441] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:07.741882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c432d0-d42f-43f1-91b9-616280deb4b1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:07.742285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c432d0-d42f-43f1-91b9-616280deb4b1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:07.742449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c432d0-d42f-43f1-91b9-616280deb4b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:07.782134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:07.807976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e1c432d0-d42f-43f1-91b9-616280deb4b1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:07.809768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e1c432d0-d42f-43f1-91b9-616280deb4b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:07.905986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 714/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:07.945420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e691525e-da8a-492d-8162-9ae36ad34bde None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:07.946963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e691525e-da8a-492d-8162-9ae36ad34bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:08.050780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e691525e-da8a-492d-8162-9ae36ad34bde None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:08.082412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:08.125357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:08.126006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:08.126293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:08.126563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:08.128194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:08.235052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:08.236476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092619d-7a50-42eb-910c-389e4f37e5c3 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:08.255082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 729/1443] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:10.423653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6783e0a-2d71-4d3b-ae60-fdd33ff8b7c0 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:10.424172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6783e0a-2d71-4d3b-ae60-fdd33ff8b7c0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:10.424285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6783e0a-2d71-4d3b-ae60-fdd33ff8b7c0 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:10.425623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 715/1444] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:10.430763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e2e9b1-fb5c-4ffd-80ae-3f8e90bdb03c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:10.431207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e2e9b1-fb5c-4ffd-80ae-3f8e90bdb03c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:10.431299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e2e9b1-fb5c-4ffd-80ae-3f8e90bdb03c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:10.494894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-59e2e9b1-fb5c-4ffd-80ae-3f8e90bdb03c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:10.496131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-59e2e9b1-fb5c-4ffd-80ae-3f8e90bdb03c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:10.508807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:10.592153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 730/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:10.630636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6783e0a-2d71-4d3b-ae60-fdd33ff8b7c0 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:10.632166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d6783e0a-2d71-4d3b-ae60-fdd33ff8b7c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:10.736167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6783e0a-2d71-4d3b-ae60-fdd33ff8b7c0 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:10.748099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:10.809041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664de754-cec8-4419-b901-20c33941128a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:10.809690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664de754-cec8-4419-b901-20c33941128a service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:10.809977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664de754-cec8-4419-b901-20c33941128a service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:10.810255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664de754-cec8-4419-b901-20c33941128a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:10.811914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-664de754-cec8-4419-b901-20c33941128a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:10.920530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664de754-cec8-4419-b901-20c33941128a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:10.922061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664de754-cec8-4419-b901-20c33941128a service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:10.945914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 716/1446] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:13.292532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc16e42-a1a3-4980-abe1-73ce468fd486 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:13.293091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc16e42-a1a3-4980-abe1-73ce468fd486 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:13.293091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc16e42-a1a3-4980-abe1-73ce468fd486 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:13.294417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 731/1447] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:13.299681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bce560-7aff-46e4-8e9f-a6e08a69ed5d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:13.300058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bce560-7aff-46e4-8e9f-a6e08a69ed5d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:13.300250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bce560-7aff-46e4-8e9f-a6e08a69ed5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:13.349958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:13.369153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d3bce560-7aff-46e4-8e9f-a6e08a69ed5d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:13.370395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d3bce560-7aff-46e4-8e9f-a6e08a69ed5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:13.465498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 717/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:13.503946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc16e42-a1a3-4980-abe1-73ce468fd486 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:13.505551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3dc16e42-a1a3-4980-abe1-73ce468fd486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:13.611984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc16e42-a1a3-4980-abe1-73ce468fd486 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:13.649927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:13.684365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:13.685035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:13.685319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:13.685592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:13.687212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:13.794169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:13.795585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d36ea2-963d-459c-8bf5-9fd1a1c771dd service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:13.812973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 732/1449] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:15.958954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627b677-44c1-4f3b-9509-3e92df7a3117 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:15.959401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627b677-44c1-4f3b-9509-3e92df7a3117 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:15.959515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627b677-44c1-4f3b-9509-3e92df7a3117 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:15.960938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 718/1450] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:15.966129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9f54fa-c256-4f8b-873a-1ecaf4b3bb45 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:15.966558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9f54fa-c256-4f8b-873a-1ecaf4b3bb45 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:15.966708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9f54fa-c256-4f8b-873a-1ecaf4b3bb45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:16.030563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-0c9f54fa-c256-4f8b-873a-1ecaf4b3bb45 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:16.031761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0c9f54fa-c256-4f8b-873a-1ecaf4b3bb45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:16.051993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:16.126789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 733/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:16.164548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627b677-44c1-4f3b-9509-3e92df7a3117 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:16.166098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f627b677-44c1-4f3b-9509-3e92df7a3117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:16.271528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627b677-44c1-4f3b-9509-3e92df7a3117 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:16.292192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:16.347492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:16.348156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:16.348446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:16.348749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:16.350443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:16.462237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:16.463671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafe9eee-2d8e-4676-861b-c624b2bff4ab service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:16.482571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 719/1452] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:18.553256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b402cf2-3ef1-49ae-a2b8-d7ad30b05100 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:18.553768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b402cf2-3ef1-49ae-a2b8-d7ad30b05100 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:18.553884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b402cf2-3ef1-49ae-a2b8-d7ad30b05100 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:18.555205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 734/1453] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:18.560916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00560287-7443-4d96-9851-80ec27392089 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:18.561351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00560287-7443-4d96-9851-80ec27392089 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:18.561498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00560287-7443-4d96-9851-80ec27392089 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:18.622276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:18.625049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-00560287-7443-4d96-9851-80ec27392089 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:18.626889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-00560287-7443-4d96-9851-80ec27392089 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:18.721587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 720/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:18.759976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b402cf2-3ef1-49ae-a2b8-d7ad30b05100 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:18.762204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5b402cf2-3ef1-49ae-a2b8-d7ad30b05100 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:18.865095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b402cf2-3ef1-49ae-a2b8-d7ad30b05100 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:18.912785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:18.938793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:18.939441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:18.939725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:18.939999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:18.941660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:19.049483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:19.050821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2eca3a-5631-4ce6-8bb6-35f3229c1138 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:19.068263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 735/1455] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:21.137214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8806758-b8cf-4cbc-a614-d735a89d7044 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:21.137766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8806758-b8cf-4cbc-a614-d735a89d7044 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:21.137766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8806758-b8cf-4cbc-a614-d735a89d7044 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:21.139165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 721/1456] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:21.144312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18e2c2e-c4ee-484c-8557-2033ffb29115 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:21.144738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18e2c2e-c4ee-484c-8557-2033ffb29115 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:21.144918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18e2c2e-c4ee-484c-8557-2033ffb29115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:21.208479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c18e2c2e-c4ee-484c-8557-2033ffb29115 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:21.209640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c18e2c2e-c4ee-484c-8557-2033ffb29115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:21.238102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:21.304967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 736/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:21.343235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8806758-b8cf-4cbc-a614-d735a89d7044 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:21.344818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c8806758-b8cf-4cbc-a614-d735a89d7044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:21.445884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8806758-b8cf-4cbc-a614-d735a89d7044 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:21.473549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:21.517528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:21.518168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:21.518451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:21.518724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:21.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:21.628090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:21.629615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddeefaf-759f-4ea0-b4f5-d3fdfaae5ad3 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:21.647637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 722/1458] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:23.764940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc678bc1-a6f1-45b6-be3b-523b819629fc None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:23.765517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc678bc1-a6f1-45b6-be3b-523b819629fc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:23.765517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc678bc1-a6f1-45b6-be3b-523b819629fc None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:23.766926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 737/1459] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:23.772056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b35e1-2db8-43f2-bf52-a8a96241393d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:23.772437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b35e1-2db8-43f2-bf52-a8a96241393d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:23.772618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b35e1-2db8-43f2-bf52-a8a96241393d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:23.837378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ed5b35e1-2db8-43f2-bf52-a8a96241393d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:23.838586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ed5b35e1-2db8-43f2-bf52-a8a96241393d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:23.846746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:23.934273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 723/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:23.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc678bc1-a6f1-45b6-be3b-523b819629fc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:23.973567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bc678bc1-a6f1-45b6-be3b-523b819629fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:24.079763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc678bc1-a6f1-45b6-be3b-523b819629fc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:24.133468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:24.152406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:24.153079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:24.153360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:24.153631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:24.155256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:24.260876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:24.262814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82551349-48e9-4ffc-98c7-7d31e051ad89 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:24.280686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 738/1461] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:26.378997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f21ac6-da7f-417e-8c9c-52722db2515c None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:26.379456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f21ac6-da7f-417e-8c9c-52722db2515c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:26.379678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f21ac6-da7f-417e-8c9c-52722db2515c None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:26.381129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 724/1462] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:26.387233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8adc902-3666-430c-9f5f-535e531df4e0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:26.387692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8adc902-3666-430c-9f5f-535e531df4e0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:26.387798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8adc902-3666-430c-9f5f-535e531df4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:26.568289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c8adc902-3666-430c-9f5f-535e531df4e0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:26.569578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c8adc902-3666-430c-9f5f-535e531df4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:26.607313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:26.664134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 739/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 10:58:26.702397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f21ac6-da7f-417e-8c9c-52722db2515c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:26.703924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-70f21ac6-da7f-417e-8c9c-52722db2515c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:26.812978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f21ac6-da7f-417e-8c9c-52722db2515c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:26.850069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:26.885804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:26.887010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:26.887289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:26.887559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:26.889227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:26.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:26.999339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e70a0d0-62d6-4993-b9e7-49249ac2ba2f service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:27.017933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 725/1464] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:29.111060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abcd14-f8e6-4cdf-a326-e9926668f1a9 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:29.111621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abcd14-f8e6-4cdf-a326-e9926668f1a9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:29.111621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abcd14-f8e6-4cdf-a326-e9926668f1a9 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:29.113012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 740/1465] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:29.118628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef71c70-c7af-4c1c-88b9-3500919f1573 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:29.119044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef71c70-c7af-4c1c-88b9-3500919f1573 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:29.119196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef71c70-c7af-4c1c-88b9-3500919f1573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:29.183952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7ef71c70-c7af-4c1c-88b9-3500919f1573 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:29.185248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7ef71c70-c7af-4c1c-88b9-3500919f1573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:29.202664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:29.282054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 726/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:29.320747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abcd14-f8e6-4cdf-a326-e9926668f1a9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:29.322302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b2abcd14-f8e6-4cdf-a326-e9926668f1a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:29.424342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abcd14-f8e6-4cdf-a326-e9926668f1a9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:29.488582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:29.495734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:29.496368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:29.496648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:29.496948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:29.498548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:29.604239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:29.605708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7304fb67-bb56-42ea-bdc4-39aecc97fa39 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:29.623248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 741/1467] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:31.634191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fbc4ec-2778-4b4f-b088-734ee7889ca1 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:31.634753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fbc4ec-2778-4b4f-b088-734ee7889ca1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:31.634753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fbc4ec-2778-4b4f-b088-734ee7889ca1 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:31.636143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 727/1468] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 10:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:31.641519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02abe876-8ac7-4e1e-850b-cb26bce28af7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:31.641933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02abe876-8ac7-4e1e-850b-cb26bce28af7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:31.642067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02abe876-8ac7-4e1e-850b-cb26bce28af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:31.705117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-02abe876-8ac7-4e1e-850b-cb26bce28af7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:31.706311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-02abe876-8ac7-4e1e-850b-cb26bce28af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:31.753837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:31.801265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 742/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 10:58:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:31.838407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fbc4ec-2778-4b4f-b088-734ee7889ca1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:31.839930 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e3fbc4ec-2778-4b4f-b088-734ee7889ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:31.942050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fbc4ec-2778-4b4f-b088-734ee7889ca1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:31.989320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:32.015031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:32.015671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:32.015949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:32.016220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:32.017875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:32.127205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:32.128636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1c064e-3b1d-4e6f-abaa-4d9697e75522 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:32.146424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 728/1470] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:41.799423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=22 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:42.144179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=66 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:51.789385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b129d82-1f41-4aa9-a46b-919cfce5cb2f None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:51.789945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b129d82-1f41-4aa9-a46b-919cfce5cb2f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:51.789945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b129d82-1f41-4aa9-a46b-919cfce5cb2f None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:51.791308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 743/1471] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:51.797071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2858b5-8dc2-462a-a8d0-0928d91b975b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:51.797475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2858b5-8dc2-462a-a8d0-0928d91b975b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:51.797663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2858b5-8dc2-462a-a8d0-0928d91b975b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:51.861043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9d2858b5-8dc2-462a-a8d0-0928d91b975b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:51.862254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9d2858b5-8dc2-462a-a8d0-0928d91b975b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:51.957845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 729/1472] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:58:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:52.040872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eca765-0490-44f7-abce-9409ad36a9e9 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:52.041112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eca765-0490-44f7-abce-9409ad36a9e9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:52.041395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eca765-0490-44f7-abce-9409ad36a9e9 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:52.042797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 744/1473] 10.222.0.33 () {54 vars in 961 bytes} [Tue Jul 19 10:58:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:52.047445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40949c37-c89c-4563-9ad4-63a23c27ab8d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:52.047737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40949c37-c89c-4563-9ad4-63a23c27ab8d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:52.048014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40949c37-c89c-4563-9ad4-63a23c27ab8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:52.121106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-40949c37-c89c-4563-9ad4-63a23c27ab8d None None] MFA Rules not processed for user `9f86b27721074978b12f34dd591362ce`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:52.122294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-40949c37-c89c-4563-9ad4-63a23c27ab8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:52.142395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:52.196454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 730/1474] 10.222.0.33 () {58 vars in 1058 bytes} [Tue Jul 19 10:58:52 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:52.204354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eca765-0490-44f7-abce-9409ad36a9e9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:52.205894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b7eca765-0490-44f7-abce-9409ad36a9e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:52.307998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eca765-0490-44f7-abce-9409ad36a9e9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:52.379241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:52.379895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:52.380176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:52.380447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:52.382085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:52.434804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:52.489668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:52.491095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002ae4eb-b269-483f-9e10-ca898e57f8e1 service glance] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:52.509717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 745/1475] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:54.628471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a6a9f6-781d-48d1-ae67-127f0a71fd8c None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:54.629069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a6a9f6-781d-48d1-ae67-127f0a71fd8c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:54.629069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a6a9f6-781d-48d1-ae67-127f0a71fd8c None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:54.630428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 731/1476] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:58:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:54.636138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef9eb36-833a-4d62-9e78-98aab1b93eab None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:54.636559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef9eb36-833a-4d62-9e78-98aab1b93eab None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:54.636778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef9eb36-833a-4d62-9e78-98aab1b93eab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:54.702312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-eef9eb36-833a-4d62-9e78-98aab1b93eab None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:54.703514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-eef9eb36-833a-4d62-9e78-98aab1b93eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:54.797349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 746/1477] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:58:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:54.821179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a6a9f6-781d-48d1-ae67-127f0a71fd8c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:54.822731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-08a6a9f6-781d-48d1-ae67-127f0a71fd8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:54.949831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a6a9f6-781d-48d1-ae67-127f0a71fd8c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:54.998575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:55.042758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:55.043416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:55.043701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:55.043976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:55.044730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:55.046542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 10:58:55.046542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 10:58:55.046542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 10:58:55.047744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5886fb9-282e-455b-bc24-800130fcc8e6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:55.066729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 732/1478] 10.222.0.33 () {56 vars in 1108 bytes} [Tue Jul 19 10:58:54 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 10:58:57.099147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57abe8e4-da59-400f-994d-790a49e37688 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:57.099670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57abe8e4-da59-400f-994d-790a49e37688 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:57.099670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57abe8e4-da59-400f-994d-790a49e37688 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:57.101099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 747/1479] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:58:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:57.106413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67ff97f-2abc-4661-adcc-e7b45a44dd41 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:57.106825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67ff97f-2abc-4661-adcc-e7b45a44dd41 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:57.106975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67ff97f-2abc-4661-adcc-e7b45a44dd41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:57.177130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b67ff97f-2abc-4661-adcc-e7b45a44dd41 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:57.178335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b67ff97f-2abc-4661-adcc-e7b45a44dd41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:57.273864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 733/1480] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:58:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:58:57.316866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57abe8e4-da59-400f-994d-790a49e37688 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:58:57.318402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-57abe8e4-da59-400f-994d-790a49e37688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:57.328556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:57.463231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57abe8e4-da59-400f-994d-790a49e37688 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:58:57.543683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:58:57.576798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:58:57.577441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:57.577723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:57.577996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:57.579616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:58:57.668660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:58:57.670454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7530f792-dc90-4329-9e6c-0a853c4bed18 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:58:57.688574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 748/1481] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:58:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:58:59.876101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016dddc9-a8d1-48f6-bb46-e3e997c6aec7 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:59.876643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016dddc9-a8d1-48f6-bb46-e3e997c6aec7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:59.876643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016dddc9-a8d1-48f6-bb46-e3e997c6aec7 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:59.877965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 734/1482] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:58:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:58:59.883166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20232f18-c6b4-4db0-b2be-17489f08468e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:58:59.883590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20232f18-c6b4-4db0-b2be-17489f08468e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:58:59.883729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20232f18-c6b4-4db0-b2be-17489f08468e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:58:59.951610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-20232f18-c6b4-4db0-b2be-17489f08468e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:58:59.952855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-20232f18-c6b4-4db0-b2be-17489f08468e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:00.000578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:00.050388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 749/1483] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:58:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:59:00.086768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016dddc9-a8d1-48f6-bb46-e3e997c6aec7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:59:00.088296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-016dddc9-a8d1-48f6-bb46-e3e997c6aec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:00.111029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:00.190937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016dddc9-a8d1-48f6-bb46-e3e997c6aec7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:59:00.261994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:59:00.262635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:00.262919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:00.263196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:00.264874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:00.337824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:00.373397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:59:00.374830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4c42d-6fc0-4c47-8817-815fe5adf087 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:59:00.392424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 735/1484] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:59:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:59:02.497846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55001d1b-b424-41c0-a4a7-be7451e8874d None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:02.498270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55001d1b-b424-41c0-a4a7-be7451e8874d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:02.498457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55001d1b-b424-41c0-a4a7-be7451e8874d None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:02.499833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 750/1485] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:59:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:59:02.505129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4bd2aa-cd83-4be0-8f29-eecc4c94de66 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:02.505550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4bd2aa-cd83-4be0-8f29-eecc4c94de66 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:02.505712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4bd2aa-cd83-4be0-8f29-eecc4c94de66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:02.569732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7f4bd2aa-cd83-4be0-8f29-eecc4c94de66 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:59:02.570933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7f4bd2aa-cd83-4be0-8f29-eecc4c94de66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:02.675960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 736/1486] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:59:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:59:02.714804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55001d1b-b424-41c0-a4a7-be7451e8874d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:59:02.716335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-55001d1b-b424-41c0-a4a7-be7451e8874d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:02.820280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55001d1b-b424-41c0-a4a7-be7451e8874d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:59:02.894796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:02.896594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:59:02.897279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:02.897742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:02.898176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:02.900012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:03.010215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:59:03.012168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c96e149-ac00-4ac0-b6ad-8bb3f72b0fbe service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:59:03.030356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 751/1487] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:59:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 10:59:05.223364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a24433-c4eb-4755-8eb0-3415c02618cf None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:05.223903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a24433-c4eb-4755-8eb0-3415c02618cf None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:05.223903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a24433-c4eb-4755-8eb0-3415c02618cf None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:05.225250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 737/1488] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:59:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:59:05.230692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037a279d-c937-4c6c-a33f-e54393af6b11 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:05.231118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037a279d-c937-4c6c-a33f-e54393af6b11 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:05.231256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037a279d-c937-4c6c-a33f-e54393af6b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:05.295289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-037a279d-c937-4c6c-a33f-e54393af6b11 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:59:05.296502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-037a279d-c937-4c6c-a33f-e54393af6b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:05.358378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:05.403551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 752/1489] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:59:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:59:05.444383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a24433-c4eb-4755-8eb0-3415c02618cf None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:59:05.445966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-33a24433-c4eb-4755-8eb0-3415c02618cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:05.482081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:05.550361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a24433-c4eb-4755-8eb0-3415c02618cf None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:59:05.621302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:59:05.621931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:05.622212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:05.622485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:05.624104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:05.701268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:05.730639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:59:05.732058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c7e51-8fa8-46aa-b052-11365587cb64 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:59:05.749943 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 738/1490] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:59:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:59:07.935709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d54ec4-f12d-478a-9e74-b9d7820757dd None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:07.936156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d54ec4-f12d-478a-9e74-b9d7820757dd None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:07.936263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d54ec4-f12d-478a-9e74-b9d7820757dd None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:07.937685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 753/1491] 10.222.0.33 () {54 vars in 926 bytes} [Tue Jul 19 10:59:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 10:59:07.943153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6050f08b-6230-4580-bc48-9410a8be1bf7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:07.943551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6050f08b-6230-4580-bc48-9410a8be1bf7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:07.943733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6050f08b-6230-4580-bc48-9410a8be1bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:08.009315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6050f08b-6230-4580-bc48-9410a8be1bf7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 10:59:08.011187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6050f08b-6230-4580-bc48-9410a8be1bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:08.107088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 739/1492] 10.222.0.33 () {58 vars in 1023 bytes} [Tue Jul 19 10:59:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 10:59:08.147646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d54ec4-f12d-478a-9e74-b9d7820757dd None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 10:59:08.149208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f3d54ec4-f12d-478a-9e74-b9d7820757dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:08.252638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d54ec4-f12d-478a-9e74-b9d7820757dd None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 10:59:08.325186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 10:59:08.325831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 10:59:08.326117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 10:59:08.326389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 10:59:08.328011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 10:59:08.337116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:08.438015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 10:59:08.439433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917bb8f8-149e-4015-8239-4ac731a3cb0d service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 10:59:08.457459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 754/1493] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 10:59:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 10:59:18.105263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=82 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 10:59:18.454643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=50 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:39.922323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:39.922906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:39.922906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:39.996506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:00:39.997758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:40.091980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 740/1494] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:00:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:00:40.100081 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:40.101032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:40.101032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:40.174503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:00:40.175626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:40.268616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 755/1495] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:00:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:00:40.273271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:40.274984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:40.286228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9729a4a3-948a-4a6e-96f4-6a5b15ae6f0d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:40.346400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:40.380394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:40.381085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:40.381369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:40.381642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:40.382360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:00:40.382664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:40.383721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:40.383721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:00:40.383721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:00:40.384615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:40.395620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 741/1496] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:00:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:00:40.404342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:40.413405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc40264-7c79-475e-8dcf-bcf5fd176b14 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:40.474830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:40.508658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:40.509335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:40.509620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:40.509896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:40.510574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:40.512307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:40.512307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:00:40.512307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:00:40.513524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:40.567647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 756/1497] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:00:40 2022] POST /identity/v3/projects => generated 379 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:00:40.572268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:40.573881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:40.606484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ede495-d97e-4b1a-8650-cdcd56cbe999 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:40.702355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:40.703015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:40.703301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:40.703570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:40.704260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:40.705292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:40.705292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:00:40.705292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:00:40.706485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:40.745835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 742/1498] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:00:40 2022] POST /identity/v3/users => generated 290 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:00:40.750095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:40.751613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:40.812331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e5e5ea-2324-4f6a-9e40-79af93f83c07 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:40.893384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:40.905084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:40.905741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:40.906018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:40.906295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:40.907011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:40.908009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:40.908009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:00:40.908009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:00:40.908913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:40.921073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 757/1499] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:00:40.925158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:40.926672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:40.938632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924c9e6-6fb0-4cf9-8598-b843d54619f5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:40.963510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:41.034260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:41.034882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.035161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.035433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.036153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:41.037185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:41.037185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:00:41.037185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:00:41.038056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2133b-e76e-4b56-8115-7348cd7a9e5d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:41.050155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 743/1500] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:00:41.054356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:41.056359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:41.089901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be1903b-871e-401b-ac29-6774b8cd59e5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:41.181845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:41.182728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.183008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.183274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] PATH_INFO: `/v3/projects/dd84f67826ea449ba1ab3994bf62b8de/users/76c8004900f74b18a6db4dc84340040c/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.204904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd84f67826ea449ba1ab3994bf62b8de, user_id=76c8004900f74b18a6db4dc84340040c, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:41.205922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:41.205922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:00:41.205922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:00:41.207841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd33ead-82c3-401d-b3fe-efe13d23a94f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:41.230820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 758/1501] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:00:41 2022] PUT /identity/v3/projects/dd84f67826ea449ba1ab3994bf62b8de/users/76c8004900f74b18a6db4dc84340040c/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:00:41.242717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e298fae9-b7d8-47b0-accb-13d5b1237201 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.243500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e298fae9-b7d8-47b0-accb-13d5b1237201 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.243805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e298fae9-b7d8-47b0-accb-13d5b1237201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.298183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e298fae9-b7d8-47b0-accb-13d5b1237201 None None] MFA Rules not processed for user `76c8004900f74b18a6db4dc84340040c`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:00:41.299331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e298fae9-b7d8-47b0-accb-13d5b1237201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:41.376232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 744/1502] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:00:41 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:00:41.389482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d5ab6-b9a5-48e3-be3b-63b978b3d66c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.389778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d5ab6-b9a5-48e3-be3b-63b978b3d66c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.390061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d5ab6-b9a5-48e3-be3b-63b978b3d66c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.391426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 759/1503] 10.222.0.33 () {54 vars in 964 bytes} [Tue Jul 19 11:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:00:41.396215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80768e17-0ecc-4530-9dd8-bc380c07cc75 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.396496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80768e17-0ecc-4530-9dd8-bc380c07cc75 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.396802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80768e17-0ecc-4530-9dd8-bc380c07cc75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.426478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:41.469841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-80768e17-0ecc-4530-9dd8-bc380c07cc75 None None] MFA Rules not processed for user `102f9668a5f948f99ac4ba6fc639c8ad`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:00:41.471056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-80768e17-0ecc-4530-9dd8-bc380c07cc75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:41.545135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 745/1504] 10.222.0.33 () {58 vars in 1061 bytes} [Tue Jul 19 11:00:41 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:00:41.553049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d5ab6-b9a5-48e3-be3b-63b978b3d66c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:41.554472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c00d5ab6-b9a5-48e3-be3b-63b978b3d66c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:41.605247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:41.660509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d5ab6-b9a5-48e3-be3b-63b978b3d66c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:41.733234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:41.733876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.734161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.734437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.736114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:41.832544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:41.833941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:41.837881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:41.852155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 760/1505] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:00:41 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:00:41.894718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708c77fb-3084-4ca5-9623-9174940bf7a8 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:41.895009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708c77fb-3084-4ca5-9623-9174940bf7a8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:41.895287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708c77fb-3084-4ca5-9623-9174940bf7a8 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:41.896569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 746/1506] 10.222.0.33 () {52 vars in 829 bytes} [Tue Jul 19 11:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:00:41.900667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:41.901910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:41.999964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d21711-d3ae-4af4-83a0-4e0183daafa9 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:42.091750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:42.092550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:42.092872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:42.093148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] PATH_INFO: `/v3/users/76c8004900f74b18a6db4dc84340040c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:42.105188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=76c8004900f74b18a6db4dc84340040c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:42.106231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:42.106231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:00:42.106231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:00:42.107437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:42.116812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:42.169284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76c8004900f74b18a6db4dc84340040c'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:42.178623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76c8004900f74b18a6db4dc84340040c'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:42.196543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76c8004900f74b18a6db4dc84340040c'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:42.197214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76c8004900f74b18a6db4dc84340040c'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:42.197547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76c8004900f74b18a6db4dc84340040c'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:42.220859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 761/1507] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:00:41 2022] DELETE /identity/v3/users/76c8004900f74b18a6db4dc84340040c => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:00:42.225288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:42.225728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:42.225839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:42.280869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:00:42.282065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:42.354797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:42.367364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 747/1508] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:00:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:00:42.376930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:42.378143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:42.476496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2afbf4-8d42-4ae3-b044-492952379da6 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:42.530944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:42.551445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:42.552085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:42.552396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:42.552669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:42.554321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:42.659025 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:42.660371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c71404-1f11-412d-95fe-f0f67eecd131 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:42.678746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 762/1509] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:00:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:00:43.029944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:00:43.031199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:00:43.043150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7da733-6caa-410f-9fde-a7eb1ecad03d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:00:43.139710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:00:43.140569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:00:43.140887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:00:43.141165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] PATH_INFO: `/v3/projects/dd84f67826ea449ba1ab3994bf62b8de` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:00:43.150914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd84f67826ea449ba1ab3994bf62b8de)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:00:43.151944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:00:43.151944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:00:43.151944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:00:43.153190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:00:43.210805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd84f67826ea449ba1ab3994bf62b8de', 'request_id': 'req-93f19afd-826a-45a5-bb81-f9f522224083'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:43.217286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd84f67826ea449ba1ab3994bf62b8de', 'request_id': 'req-93f19afd-826a-45a5-bb81-f9f522224083'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:43.226358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd84f67826ea449ba1ab3994bf62b8de', 'request_id': 'req-93f19afd-826a-45a5-bb81-f9f522224083'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:43.256349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] The token cache is being invalidate because project dd84f67826ea449ba1ab3994bf62b8de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:00:43.256634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:43.257174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-93f19afd-826a-45a5-bb81-f9f522224083 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:00:43.258514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 748/1510] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:00:43 2022] DELETE /identity/v3/projects/dd84f67826ea449ba1ab3994bf62b8de => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:00:52.675999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=62 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:00:53.254255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:02:22.007210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582798a6-f497-4b06-a9ca-bd0c070964f0 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:22.007771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582798a6-f497-4b06-a9ca-bd0c070964f0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:22.007771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582798a6-f497-4b06-a9ca-bd0c070964f0 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:22.009182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 763/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 11:02:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:22.014206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00522e0c-0d76-4274-ba6d-75b3c188fdc8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:22.014652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00522e0c-0d76-4274-ba6d-75b3c188fdc8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:22.014777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00522e0c-0d76-4274-ba6d-75b3c188fdc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:22.094015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-00522e0c-0d76-4274-ba6d-75b3c188fdc8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:22.095212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-00522e0c-0d76-4274-ba6d-75b3c188fdc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:22.195054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 749/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 11:02:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:22.349042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582798a6-f497-4b06-a9ca-bd0c070964f0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:22.351125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-582798a6-f497-4b06-a9ca-bd0c070964f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:22.467555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582798a6-f497-4b06-a9ca-bd0c070964f0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:22.473018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d695c888-1075-42a6-931c-2d4a0233e7be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:22.473662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d695c888-1075-42a6-931c-2d4a0233e7be service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:22.473948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d695c888-1075-42a6-931c-2d4a0233e7be service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:22.474222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d695c888-1075-42a6-931c-2d4a0233e7be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:22.482179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d695c888-1075-42a6-931c-2d4a0233e7be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:22.483739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d695c888-1075-42a6-931c-2d4a0233e7be service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:22.492893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 764/1513] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:23.129337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ce154-b87d-4c4d-bddb-5717840f987a None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:23.129888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ce154-b87d-4c4d-bddb-5717840f987a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:23.130227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ce154-b87d-4c4d-bddb-5717840f987a None None] PATH_INFO: `/v3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:23.131645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 750/1514] 10.222.0.27 () {54 vars in 925 bytes} [Tue Jul 19 11:02:23 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:02:23.140741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da8ad59-0f34-4c67-a434-97de0ddff18b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:23.141184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da8ad59-0f34-4c67-a434-97de0ddff18b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:23.141306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da8ad59-0f34-4c67-a434-97de0ddff18b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:23.200021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9da8ad59-0f34-4c67-a434-97de0ddff18b None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:23.201172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9da8ad59-0f34-4c67-a434-97de0ddff18b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:23.269800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:02:23.276288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 765/1515] 10.222.0.27 () {58 vars in 1015 bytes} [Tue Jul 19 11:02:23 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:23.319282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebf16a-a4f3-4392-b8d5-3f7ea0ac69ef None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:23.319576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebf16a-a4f3-4392-b8d5-3f7ea0ac69ef None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:23.319859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebf16a-a4f3-4392-b8d5-3f7ea0ac69ef None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:23.321297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 751/1516] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 11:02:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:23.326028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5667fd02-76a5-4d18-a543-c931a2969370 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:23.326322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5667fd02-76a5-4d18-a543-c931a2969370 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:23.326603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5667fd02-76a5-4d18-a543-c931a2969370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:23.344529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5667fd02-76a5-4d18-a543-c931a2969370 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:23.345743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5667fd02-76a5-4d18-a543-c931a2969370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:23.354130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 766/1517] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 11:02:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:23.502891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebf16a-a4f3-4392-b8d5-3f7ea0ac69ef None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:23.517236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebf16a-a4f3-4392-b8d5-3f7ea0ac69ef None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:23.522759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf62e1-e573-475e-a53c-8d696d0864d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:23.523394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf62e1-e573-475e-a53c-8d696d0864d0 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:23.523681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf62e1-e573-475e-a53c-8d696d0864d0 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:23.523958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf62e1-e573-475e-a53c-8d696d0864d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:23.532765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf62e1-e573-475e-a53c-8d696d0864d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:23.534362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf62e1-e573-475e-a53c-8d696d0864d0 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:23.544442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 752/1518] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:02:33.339443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:02:33.539538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=75 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:02:34.748489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044ddcf-7af9-4812-b4a8-042c0f739c79 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:34.749071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044ddcf-7af9-4812-b4a8-042c0f739c79 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:34.749071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044ddcf-7af9-4812-b4a8-042c0f739c79 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:34.750454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 767/1519] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 11:02:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:34.755330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ceca64-1727-4e97-b3ef-5aafc40ddafb None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:34.755739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ceca64-1727-4e97-b3ef-5aafc40ddafb None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:34.755894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ceca64-1727-4e97-b3ef-5aafc40ddafb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:34.774314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d4ceca64-1727-4e97-b3ef-5aafc40ddafb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:34.775677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d4ceca64-1727-4e97-b3ef-5aafc40ddafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:34.783579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 753/1520] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 11:02:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:34.940262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044ddcf-7af9-4812-b4a8-042c0f739c79 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:34.954739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044ddcf-7af9-4812-b4a8-042c0f739c79 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:34.960583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52cfdeb-c634-4344-99e1-c19e6034cc39 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:34.961233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52cfdeb-c634-4344-99e1-c19e6034cc39 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:34.961514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52cfdeb-c634-4344-99e1-c19e6034cc39 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:34.961784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52cfdeb-c634-4344-99e1-c19e6034cc39 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:34.969670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52cfdeb-c634-4344-99e1-c19e6034cc39 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:34.971221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52cfdeb-c634-4344-99e1-c19e6034cc39 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:34.980979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 768/1521] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:35.540461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873c0cf-a294-4f26-b3c2-d737a6982b91 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:35.541310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873c0cf-a294-4f26-b3c2-d737a6982b91 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:35.541625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873c0cf-a294-4f26-b3c2-d737a6982b91 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:35.543039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 754/1522] 10.222.0.33 () {54 vars in 961 bytes} [Tue Jul 19 11:02:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:35.548086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc16be-9b8f-42cc-9df8-a732824a2555 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:35.548379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc16be-9b8f-42cc-9df8-a732824a2555 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:35.548657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc16be-9b8f-42cc-9df8-a732824a2555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:35.566621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ecdc16be-9b8f-42cc-9df8-a732824a2555 None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:35.567800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ecdc16be-9b8f-42cc-9df8-a732824a2555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:35.576673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 769/1523] 10.222.0.33 () {58 vars in 1058 bytes} [Tue Jul 19 11:02:35 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:35.584325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873c0cf-a294-4f26-b3c2-d737a6982b91 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:35.598202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873c0cf-a294-4f26-b3c2-d737a6982b91 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:35.603223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb95076-910b-4f04-8e8d-14425e0c8344 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:35.604355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb95076-910b-4f04-8e8d-14425e0c8344 service placement] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:35.604643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb95076-910b-4f04-8e8d-14425e0c8344 service placement] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:35.604952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb95076-910b-4f04-8e8d-14425e0c8344 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:35.613189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb95076-910b-4f04-8e8d-14425e0c8344 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:35.614770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb95076-910b-4f04-8e8d-14425e0c8344 service placement] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:35.624649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 755/1524] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:35 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:35.797456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f73c648-410a-4cd9-9511-d7cfbb4477e9 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:35.797983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f73c648-410a-4cd9-9511-d7cfbb4477e9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:35.798088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f73c648-410a-4cd9-9511-d7cfbb4477e9 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:35.799508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 770/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 19 11:02:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:35.804342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865e848f-e6ce-4e97-9cdd-3c61584966eb None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:35.804781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865e848f-e6ce-4e97-9cdd-3c61584966eb None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:35.804933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865e848f-e6ce-4e97-9cdd-3c61584966eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:35.823338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-865e848f-e6ce-4e97-9cdd-3c61584966eb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:35.824527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-865e848f-e6ce-4e97-9cdd-3c61584966eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:35.832491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 756/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 19 11:02:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:35.988661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f73c648-410a-4cd9-9511-d7cfbb4477e9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:36.003858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f73c648-410a-4cd9-9511-d7cfbb4477e9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:36.009978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6897342-ead0-455e-ac0d-a70223a483cf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:36.010606 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6897342-ead0-455e-ac0d-a70223a483cf service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:36.010893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6897342-ead0-455e-ac0d-a70223a483cf service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:36.011163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6897342-ead0-455e-ac0d-a70223a483cf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:36.019214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6897342-ead0-455e-ac0d-a70223a483cf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:36.020814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6897342-ead0-455e-ac0d-a70223a483cf service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:36.030066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 771/1527] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:40.059427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b8cb01-29c5-443e-a39c-f5de91a9ab01 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:40.059979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b8cb01-29c5-443e-a39c-f5de91a9ab01 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:40.059979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b8cb01-29c5-443e-a39c-f5de91a9ab01 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:40.061408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 757/1528] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 11:02:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:40.068128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2409de-8101-42f9-84a5-efb7737b25c4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:40.068591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2409de-8101-42f9-84a5-efb7737b25c4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:40.068712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2409de-8101-42f9-84a5-efb7737b25c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:40.087210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cb2409de-8101-42f9-84a5-efb7737b25c4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:40.088404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cb2409de-8101-42f9-84a5-efb7737b25c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:40.096583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 772/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 11:02:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:40.169733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b8cb01-29c5-443e-a39c-f5de91a9ab01 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:40.171445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-15b8cb01-29c5-443e-a39c-f5de91a9ab01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:40.249098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b8cb01-29c5-443e-a39c-f5de91a9ab01 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:40.252601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6421dc6a-dca0-47d3-aac2-5dd1f82dc865 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:40.253274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6421dc6a-dca0-47d3-aac2-5dd1f82dc865 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:40.253560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6421dc6a-dca0-47d3-aac2-5dd1f82dc865 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:40.253838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6421dc6a-dca0-47d3-aac2-5dd1f82dc865 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:40.262965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6421dc6a-dca0-47d3-aac2-5dd1f82dc865 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:40.264391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6421dc6a-dca0-47d3-aac2-5dd1f82dc865 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:40.273706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 758/1530] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:42.285616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a3bdc7-38ed-42db-a1d3-484c42d824ab None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:42.286137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a3bdc7-38ed-42db-a1d3-484c42d824ab None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:42.286137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a3bdc7-38ed-42db-a1d3-484c42d824ab None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:42.287451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 773/1531] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 11:02:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:42.293413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a87869-1c75-425b-912c-6d92c1e6f4f1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:42.293834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a87869-1c75-425b-912c-6d92c1e6f4f1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:42.293986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a87869-1c75-425b-912c-6d92c1e6f4f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:42.313834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-36a87869-1c75-425b-912c-6d92c1e6f4f1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:42.315073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-36a87869-1c75-425b-912c-6d92c1e6f4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:42.322822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 759/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 11:02:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:42.359418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a3bdc7-38ed-42db-a1d3-484c42d824ab None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:42.369964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a3bdc7-38ed-42db-a1d3-484c42d824ab None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:42.373704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fdd9d2-51d7-4506-a087-b67aaa11e22d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:42.374321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fdd9d2-51d7-4506-a087-b67aaa11e22d service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:42.374601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fdd9d2-51d7-4506-a087-b67aaa11e22d service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:42.374873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fdd9d2-51d7-4506-a087-b67aaa11e22d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:42.383570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fdd9d2-51d7-4506-a087-b67aaa11e22d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:42.385489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fdd9d2-51d7-4506-a087-b67aaa11e22d service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:42.394613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 774/1533] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:46.080538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f3b0b-1dbc-4d60-a73c-e66d5ce4f7d8 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:46.081128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f3b0b-1dbc-4d60-a73c-e66d5ce4f7d8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:46.081128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f3b0b-1dbc-4d60-a73c-e66d5ce4f7d8 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:46.082501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 760/1534] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 11:02:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:46.088228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4773f5-0efa-4879-938b-6e010ec24fc3 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:46.088805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4773f5-0efa-4879-938b-6e010ec24fc3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:46.088906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4773f5-0efa-4879-938b-6e010ec24fc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:46.106385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5d4773f5-0efa-4879-938b-6e010ec24fc3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:46.107579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5d4773f5-0efa-4879-938b-6e010ec24fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:46.115344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 775/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 11:02:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:46.423274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f3b0b-1dbc-4d60-a73c-e66d5ce4f7d8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:46.424977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-662f3b0b-1dbc-4d60-a73c-e66d5ce4f7d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:46.501638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f3b0b-1dbc-4d60-a73c-e66d5ce4f7d8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:46.505155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92518526-01fa-4f78-9a04-a62d276f4402 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:46.505784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92518526-01fa-4f78-9a04-a62d276f4402 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:46.506067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92518526-01fa-4f78-9a04-a62d276f4402 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:46.506341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92518526-01fa-4f78-9a04-a62d276f4402 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:46.515187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92518526-01fa-4f78-9a04-a62d276f4402 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:46.516600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92518526-01fa-4f78-9a04-a62d276f4402 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:46.526320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 761/1536] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:48.570329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c317c27-af52-4b6e-9e69-d5d4ba0212fe None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:48.570894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c317c27-af52-4b6e-9e69-d5d4ba0212fe None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:48.570894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c317c27-af52-4b6e-9e69-d5d4ba0212fe None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:48.572274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 776/1537] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 19 11:02:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:02:48.577484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ec42c-df23-46b8-adc5-6201da0ca0c0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:48.577880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ec42c-df23-46b8-adc5-6201da0ca0c0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:48.578042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ec42c-df23-46b8-adc5-6201da0ca0c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:48.595840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f10ec42c-df23-46b8-adc5-6201da0ca0c0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:02:48.597065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f10ec42c-df23-46b8-adc5-6201da0ca0c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:48.605380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 762/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 19 11:02:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:02:48.917480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c317c27-af52-4b6e-9e69-d5d4ba0212fe None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:02:48.919161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8c317c27-af52-4b6e-9e69-d5d4ba0212fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:02:48.932113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c317c27-af52-4b6e-9e69-d5d4ba0212fe None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:02:48.935689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f7527a-c296-47e8-9e19-d37b83497943 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:02:48.936323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f7527a-c296-47e8-9e19-d37b83497943 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:02:48.936609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f7527a-c296-47e8-9e19-d37b83497943 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:02:48.936912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f7527a-c296-47e8-9e19-d37b83497943 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:02:48.945902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f7527a-c296-47e8-9e19-d37b83497943 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:02:48.947326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f7527a-c296-47e8-9e19-d37b83497943 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:02:48.956625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 777/1539] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:02:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:02:58.591077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:02:58.952456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=33 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:19.753974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88be7f4-27e8-4643-903c-cc4b7a09a269 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.776314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88be7f4-27e8-4643-903c-cc4b7a09a269 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.776314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88be7f4-27e8-4643-903c-cc4b7a09a269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.777265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ef9e7f-e68f-40f0-a1dd-78a3eeefcf0a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.777265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ef9e7f-e68f-40f0-a1dd-78a3eeefcf0a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.777265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ef9e7f-e68f-40f0-a1dd-78a3eeefcf0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.779241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c88be7f4-27e8-4643-903c-cc4b7a09a269 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:19.779241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c88be7f4-27e8-4643-903c-cc4b7a09a269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:19.795585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 763/1540] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:19.802604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-84ef9e7f-e68f-40f0-a1dd-78a3eeefcf0a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:19.804009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-84ef9e7f-e68f-40f0-a1dd-78a3eeefcf0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:19.805007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.805307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.805588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.817320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 778/1541] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:04:19.823887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.824182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.824465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.839352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:19.841015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:19.850245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:19.852125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:19.859372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 764/1542] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:04:19.863853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:19.870620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 779/1543] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:19.875442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:19.883871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7831e5-b34b-4eee-aac9-8494b779177c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:19.889539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54ba8d1-992c-4e6d-bdf3-423a48da395b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:19.892328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:19.893031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.893469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.893753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.894497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:19.894617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:19.894815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:19.895166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.895460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.895740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.896476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:19.896668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:19.896806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:19.898616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:04:19.898616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:04:19.898616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:04:19.899499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:19.920887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 765/1544] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:04:19.928738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 780/1545] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:04:19.928964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:19.933004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:19.946177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c56b88-0d34-4c41-8a33-36cf7bc1fbbb admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:19.946749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27b09a-351e-4b20-8a32-237421fc2d4a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:19.957437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:19.958529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.959022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.959529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.961112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:19.963367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:19.968727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:19.968727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:19.968727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:19.968727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:19.969152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:19.969152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:04:19.969152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:04:19.969152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:04:19.972721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511880e3-8dc2-4c1c-ab09-6e390f75936e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.030849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 766/1546] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:19 2022] POST /identity/v3/projects => generated 393 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.032123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93d82c2-bdb8-42a7-a303-c054480d789e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.032419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93d82c2-bdb8-42a7-a303-c054480d789e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.032721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93d82c2-bdb8-42a7-a303-c054480d789e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.039244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 781/1547] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:19 2022] POST /identity/v3/projects => generated 379 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.040372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.061832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b93d82c2-bdb8-42a7-a303-c054480d789e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.063061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b93d82c2-bdb8-42a7-a303-c054480d789e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.144330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 767/1548] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.146558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.146857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.147185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.158288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2305db4-7958-4913-86e2-8c99ec8aa8c3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.164190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.164905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.165240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.165523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.166156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.167167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:04:20.167167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:04:20.167167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:04:20.168376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70440cab-51e3-4f7e-b224-8327283e0315 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.169344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.170601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.179762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 768/1549] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.180545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.195053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1329a5d2-3a9d-4fee-aef9-102cc845266b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.199559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 782/1550] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/users => generated 297 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.200476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24b2c1-f722-4d01-a85a-8f02c434200d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.200966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d917b-cc47-4336-9787-948ca25eca54 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.201290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d917b-cc47-4336-9787-948ca25eca54 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.201411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24b2c1-f722-4d01-a85a-8f02c434200d admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.201570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d917b-cc47-4336-9787-948ca25eca54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.201697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24b2c1-f722-4d01-a85a-8f02c434200d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.201900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24b2c1-f722-4d01-a85a-8f02c434200d admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.202725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24b2c1-f722-4d01-a85a-8f02c434200d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.204752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24b2c1-f722-4d01-a85a-8f02c434200d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.222312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-293d917b-cc47-4336-9787-948ca25eca54 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.223532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-293d917b-cc47-4336-9787-948ca25eca54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.226251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 769/1551] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/users => generated 290 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.227520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ab2e8-f86b-4f1d-86c3-0b8af76689db None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.227812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ab2e8-f86b-4f1d-86c3-0b8af76689db None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.228091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ab2e8-f86b-4f1d-86c3-0b8af76689db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.237215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 783/1552] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.238803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d6c86b-bbc8-462a-9d86-d30f1375c88e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.239368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d6c86b-bbc8-462a-9d86-d30f1375c88e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.239979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d6c86b-bbc8-462a-9d86-d30f1375c88e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.247591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e49ab2e8-f86b-4f1d-86c3-0b8af76689db None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.249013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e49ab2e8-f86b-4f1d-86c3-0b8af76689db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.258955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 770/1553] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.260395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.260826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.261730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.264264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-80d6c86b-bbc8-462a-9d86-d30f1375c88e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.265663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-80d6c86b-bbc8-462a-9d86-d30f1375c88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.277921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 784/1554] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.277921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.277921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.277921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.286240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.287525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.296082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 771/1555] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.297207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.298760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.299947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.307764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99063d0b-d5c4-422c-b712-e7ca6d75b715 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.308433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 785/1556] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.309260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.312599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0eb884b-cd15-4929-9062-37bd289ca0b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.313282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eb884b-cd15-4929-9062-37bd289ca0b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.313631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eb884b-cd15-4929-9062-37bd289ca0b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.313916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eb884b-cd15-4929-9062-37bd289ca0b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.314678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eb884b-cd15-4929-9062-37bd289ca0b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.316255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eb884b-cd15-4929-9062-37bd289ca0b6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.318728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbca504-9d99-43c8-ab3c-d50f019c1e2c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.323977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a59e5f-73f3-4cf1-91ad-236f967d9eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.324625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a59e5f-73f3-4cf1-91ad-236f967d9eb6 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.324925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a59e5f-73f3-4cf1-91ad-236f967d9eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.325195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a59e5f-73f3-4cf1-91ad-236f967d9eb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.325914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a59e5f-73f3-4cf1-91ad-236f967d9eb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.327328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 772/1557] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:20.327477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a59e5f-73f3-4cf1-91ad-236f967d9eb6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.328381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.328676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.328979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.340537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 786/1558] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:20.340537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae98667-2445-4fb5-afaa-0fddd0a1cb92 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.340537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae98667-2445-4fb5-afaa-0fddd0a1cb92 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.340537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae98667-2445-4fb5-afaa-0fddd0a1cb92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.346806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.348018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.355981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 773/1559] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.356761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.360614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-3ae98667-2445-4fb5-afaa-0fddd0a1cb92 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.361850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3ae98667-2445-4fb5-afaa-0fddd0a1cb92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.365977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6523b55e-7d20-413a-90d0-2743acf665d7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.370186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 787/1560] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.371041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.371628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.371914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.372013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.372191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.372285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.372460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.373218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:20.373532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.375112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7e2ff-3881-4ece-be3d-6d41a3620da2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.387726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 774/1561] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:20.388843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.389130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.389408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.390498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.391678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.399587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 788/1562] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.400362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.407130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.408321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.410151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60001a25-50f7-4b5c-9c25-b522fed229a6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.414641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.415280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.415563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.415841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.416544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 775/1563] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.416656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:20.417008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.417221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.418616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.426303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475add46-2ebf-4195-8a20-a9e7a75da1d9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.431419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 789/1564] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:20.431554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.432026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.432154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.432334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.432572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.433317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.434975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.441546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33659c77-37b6-4372-9f67-22090cc47e7e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.446821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 776/1565] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:20.446821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.448771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.449489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.449781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.450058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.450789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.452433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.456988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21162c78-4774-4f60-9de8-160a17a34200 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.462133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.462774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.463058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.463357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.464082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:20.464211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 790/1566] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:20.464386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.465024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.466007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.476398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b3bc95-6cb8-4bc9-9db1-9d177ea0ac01 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.480245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 777/1567] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:20.481129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.481263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.481901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.482192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.482470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.483194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:20.483505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.485140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1252b879-6446-47a5-8915-46505ab63b18 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.488431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:20.492404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0b8729-e330-42fb-8714-1a0354c89a02 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.498503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 791/1568] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:20.499057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.499782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.499957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.500209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.500464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.500648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.500933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.502094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.504142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.518445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:20.519715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:20.528655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 792/1569] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:20.529289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.534700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 778/1570] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.535575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.539330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52efc9-c727-4522-9539-0df9862deb9d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.540141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:20.601985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.602618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5812d6-9132-4eaa-80c5-053a959df1fd admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.602742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.602922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.603196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.603916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:20.604223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.605901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.607838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.609041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.609328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.609599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.610353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.612247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.617721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 793/1571] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:20.618517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.630806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65767f8-fc9c-49fd-8596-9124426a4a82 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.636312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.637704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.637704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.637926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] PATH_INFO: `/v3/projects/05fceba2c70c40a399e71daabe8edef6/users/c95281b28eae450d8678583a929c3b8a/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.645090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 779/1572] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/projects => generated 383 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.645090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.654233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05fceba2c70c40a399e71daabe8edef6, user_id=c95281b28eae450d8678583a929c3b8a, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.656998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.669954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 794/1573] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:20 2022] PUT /identity/v3/projects/05fceba2c70c40a399e71daabe8edef6/users/c95281b28eae450d8678583a929c3b8a/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:20.670637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.712643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fcf8dc-53ac-4d42-9c40-937f41d18b69 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.737321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b070c660-e13e-47d5-aff9-9375680ec664 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.742002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.742655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.742942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.743217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.743909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.745866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.770042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.770939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.771221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.771496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] PATH_INFO: `/v3/projects/205683784b714ab399311aeca15d0cfa/users/f6cb55e4c41f4e6d94156c597a1baad8/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.772895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 795/1574] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/projects => generated 367 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.777573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.787219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=205683784b714ab399311aeca15d0cfa, user_id=f6cb55e4c41f4e6d94156c597a1baad8, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.788319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:04:20.788319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:04:20.788319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:04:20.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.798552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:20.804211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 780/1575] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:20 2022] PUT /identity/v3/projects/205683784b714ab399311aeca15d0cfa/users/f6cb55e4c41f4e6d94156c597a1baad8/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:20.805236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.844462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4fe99e-94b3-4681-bf0e-91411682e6e9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.848205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:20.870301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52194da-75c0-48f5-a004-fd116c093f65 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.875269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.875932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.876223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.876509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.877273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:20.877583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.879179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.891203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 781/1576] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:20.891996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.902193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce9bcef-1d91-458c-9f8e-a2e31a65e939 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.903660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.904324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.904614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.904989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.905685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.906874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:20.907523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:20.907646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.907826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:20.908083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:20.908802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:20.910692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:20.930734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 782/1577] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/users => generated 293 bytes in 39 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.931617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.935849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 796/1578] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/projects => generated 385 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:20.936850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:20.944368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2456bb0-dd21-454d-8785-8439af2654a3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:20.999827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe6c7ae-b886-474a-863e-10da79076b87 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.003420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.004091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.004384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.004674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.005391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.005524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9d11a9-577e-40e3-9f61-2317da8bfe1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.006166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9d11a9-577e-40e3-9f61-2317da8bfe1b admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.006450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9d11a9-577e-40e3-9f61-2317da8bfe1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.006728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9d11a9-577e-40e3-9f61-2317da8bfe1b admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.007372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.007490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9d11a9-577e-40e3-9f61-2317da8bfe1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.009367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9d11a9-577e-40e3-9f61-2317da8bfe1b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.030857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 797/1579] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/users => generated 292 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.032130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f5468-f128-4b31-b6cd-5f3181ff881c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.032427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f5468-f128-4b31-b6cd-5f3181ff881c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.032727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f5468-f128-4b31-b6cd-5f3181ff881c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.035595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 783/1580] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:20 2022] POST /identity/v3/projects => generated 373 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.036623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.051073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ce8f5468-f128-4b31-b6cd-5f3181ff881c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:21.052225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ce8f5468-f128-4b31-b6cd-5f3181ff881c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:21.103715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27104431-dd9a-48c9-9bff-94a3aae04344 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.108846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.109519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.109809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.110092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.110418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 798/1581] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:21.110786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.111686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.111974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.112249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.112736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.119503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:21.130882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:21.132371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:21.141177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 784/1582] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/users => generated 284 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.141177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.141410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 799/1583] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:21.143847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.149469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f9f52-85aa-46a9-abf5-34201be4450d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.153883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ead5b1d-07a2-4426-bb56-77379afcd958 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.154681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.155382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.155652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.155928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.156618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.158534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.158780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.159181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.159492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.159768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.160490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.162106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.175819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 800/1584] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.175819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.182052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:21.187143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce4488-ffaa-4e53-9df3-865e6e1895fe admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.190729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 785/1585] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.190729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.245792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.246714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.247141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.247413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.248098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.250091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.252050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7136d2-1283-4e90-a171-e9a6477c09b5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.257177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.257845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.258162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.258441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.259169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.260976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.270774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 801/1586] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.271573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.272126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 786/1587] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.273663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.283435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07b5116-5365-4d13-855e-0cb69d2527eb admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.288179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.289447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.289802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.290119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.290802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.292729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.311908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 802/1588] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/users => generated 287 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.312647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.322122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe3ccbf-8b6c-4732-968c-54a605e4823c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.326755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.327394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.327681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.327963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.328681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.330288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.330746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eff438-ac05-4ebe-a333-6a60b880bb8a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.336966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.337732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.338029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.338450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.341409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 803/1589] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.342366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.351061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.352275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9bcb8c-7419-4124-9ef8-c86969be8e41 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.352485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.355967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.356583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.356919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.357195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.362258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 787/1590] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:21.363139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.367205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.368942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.373186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1101f8-bd7a-41e8-8b8c-bec04fc20850 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.378414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.379098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.379388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.379695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.379815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 804/1591] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:21.380564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.380687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.382718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.390897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd6dab-b558-426c-afb2-865c65fde51d service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.394131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 788/1592] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.395098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.399244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.400177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.400177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.400448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.401187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.403096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.405356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8932452-0b1e-498e-9d8c-b20d5b7a25ba admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.411008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.411833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.412122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.412510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.413985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.416210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.427424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 805/1593] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/users => generated 293 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:21.431300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 789/1594] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.431931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.432196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.442177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419bf2b-3fbb-4183-9fbe-5550ee3f7bf9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.442809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ae711-c30d-4d9b-a3e1-4b2c18e331ee admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.448232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.449473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.449905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.450105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.450483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.450844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.451296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.451583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.451862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.452518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.453200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.454861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.466829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 806/1595] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.466829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.478849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856d2477-40a7-41e4-b1d9-781bd6593a71 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.484304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.485149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.485509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.485869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.486733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.488460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.502682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 807/1596] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.503510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.514375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a132e4-3808-4013-915f-24847b8cc0af admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.520777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.521917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.522301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.522750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] PATH_INFO: `/v3/projects/e1d59596ecae4e288733b90c577aa86b/users/246dc188d9a44b1cb38721e3dd8a863a/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.542272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1d59596ecae4e288733b90c577aa86b, user_id=246dc188d9a44b1cb38721e3dd8a863a, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.545546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.561568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 808/1597] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:21 2022] PUT /identity/v3/projects/e1d59596ecae4e288733b90c577aa86b/users/246dc188d9a44b1cb38721e3dd8a863a/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:21.563035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.601849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 790/1598] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.601849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.610918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af739890-5bd1-4a18-906d-678c2c9692c5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.629920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:21.650529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37512fd5-26bf-45d9-8d05-878e23ced336 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.652772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:21.656158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.656998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.657367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.657750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.658623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.660737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.672966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 809/1599] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.673645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.683951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7118ee9d-fd77-475f-8fb8-d6edb77c2ead admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.688955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06f7886-60ca-499d-8f4b-d3e5aeb29161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.689862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06f7886-60ca-499d-8f4b-d3e5aeb29161 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.690153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06f7886-60ca-499d-8f4b-d3e5aeb29161 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.690456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06f7886-60ca-499d-8f4b-d3e5aeb29161 admin admin] PATH_INFO: `/v3/projects/ef534f1d06ea415d98857ec7f9e5fb97/users/b8f2a78f3a3e4b8e8937e3bc39524690/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.698411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.699337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.699777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.700664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.701761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.703603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.708639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06f7886-60ca-499d-8f4b-d3e5aeb29161 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef534f1d06ea415d98857ec7f9e5fb97, user_id=b8f2a78f3a3e4b8e8937e3bc39524690, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.712112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06f7886-60ca-499d-8f4b-d3e5aeb29161 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.718467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 791/1600] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.718467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.727336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72442e9c-1d4e-4da1-91f7-f4c26e88ca97 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.732334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 810/1601] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:21 2022] PUT /identity/v3/projects/ef534f1d06ea415d98857ec7f9e5fb97/users/b8f2a78f3a3e4b8e8937e3bc39524690/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:21.734103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facc785c-e754-4fde-ad75-5b791289209d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.734404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facc785c-e754-4fde-ad75-5b791289209d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.734682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facc785c-e754-4fde-ad75-5b791289209d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.756957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-facc785c-e754-4fde-ad75-5b791289209d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:21.759019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-facc785c-e754-4fde-ad75-5b791289209d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:21.818068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.818971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.819256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.819577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] PATH_INFO: `/v3/projects/d6f07e7df33a48d7a9820f4100e7caee/users/d6d30fa2b0274f7087c8ba3394bafb44/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.835764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6f07e7df33a48d7a9820f4100e7caee, user_id=d6d30fa2b0274f7087c8ba3394bafb44, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.836301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 811/1602] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:21.837275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facc785c-e754-4fde-ad75-5b791289209d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.841702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.849387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facc785c-e754-4fde-ad75-5b791289209d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.856558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.857555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.857849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.858154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.858893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.860880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 792/1603] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:21 2022] PUT /identity/v3/projects/d6f07e7df33a48d7a9820f4100e7caee/users/d6d30fa2b0274f7087c8ba3394bafb44/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:21.861028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.861309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.874476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 812/1604] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.875436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:21.886339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1340baa7-80dd-4354-ab5e-d2aa751df1c1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.964518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e96f9-c3ce-44d0-888a-5a7650ae980e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.965608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74603edd-05bb-455c-bcff-53c82427dbab admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:21.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e96f9-c3ce-44d0-888a-5a7650ae980e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e96f9-c3ce-44d0-888a-5a7650ae980e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.965933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e96f9-c3ce-44d0-888a-5a7650ae980e admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.966641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e96f9-c3ce-44d0-888a-5a7650ae980e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.968611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e96f9-c3ce-44d0-888a-5a7650ae980e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:21.972025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:21.972957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.973246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.974078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] PATH_INFO: `/v3/projects/479a805c7ec24812ba05c59b6defa408/users/bd202e0ca919481583fb2559be68eee0/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.981176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 813/1605] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:21.982267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ddd9b-55a0-43c2-938d-ebfd38c5e74b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:21.982563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ddd9b-55a0-43c2-938d-ebfd38c5e74b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:21.982842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ddd9b-55a0-43c2-938d-ebfd38c5e74b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:21.993432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] RBAC: Authorizing `identity:create_grant(project_id=479a805c7ec24812ba05c59b6defa408, user_id=bd202e0ca919481583fb2559be68eee0, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:21.996150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:22.005906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-099ddd9b-55a0-43c2-938d-ebfd38c5e74b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:22.007200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-099ddd9b-55a0-43c2-938d-ebfd38c5e74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:22.017073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 793/1606] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:21 2022] PUT /identity/v3/projects/479a805c7ec24812ba05c59b6defa408/users/bd202e0ca919481583fb2559be68eee0/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:22.018308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:22.020611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:22.102272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:22.116207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:22.146589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b304e3-c21f-4945-9f24-a446be53249b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:22.147636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 814/1607] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:22.149407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.149798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.150280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.157213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:22.158074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.159301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.159654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.187501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:22.189009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] MFA Rules not processed for user `d6d30fa2b0274f7087c8ba3394bafb44`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:22.189167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:22.191032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:22.220555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 794/1608] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:22.221484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:22.242204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea7084e-f90a-4965-b104-85047a1fd840 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:22.261035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf720c58-9029-4400-aa82-6d157bc55cb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:22.262268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf720c58-9029-4400-aa82-6d157bc55cb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.262842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf720c58-9029-4400-aa82-6d157bc55cb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.263102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf720c58-9029-4400-aa82-6d157bc55cb6 admin admin] PATH_INFO: `/v3/projects/7d0d678908014f8dae55bc6475b84680/users/e68eb1137cc845148ba17a3d5c46f878/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.294764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf720c58-9029-4400-aa82-6d157bc55cb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d0d678908014f8dae55bc6475b84680, user_id=e68eb1137cc845148ba17a3d5c46f878, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:22.303343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 815/1609] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:22 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:22.303686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf720c58-9029-4400-aa82-6d157bc55cb6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:22.304436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:22.322042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba1a15b-875d-48e5-845b-539253a72cc5 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:22.338462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 795/1610] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:22 2022] PUT /identity/v3/projects/7d0d678908014f8dae55bc6475b84680/users/e68eb1137cc845148ba17a3d5c46f878/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:22.354501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.354955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.356562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.411887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] MFA Rules not processed for user `bd202e0ca919481583fb2559be68eee0`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:22.413897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:22.494506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:22.498185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.498508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.498794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] PATH_INFO: `/v3/projects/888be6c04e3e4248a000a1c601a1c8ed/users/2c9c1fff948941b8baacac536fc09724/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.515124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 796/1611] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:22.516091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:22.545211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=888be6c04e3e4248a000a1c601a1c8ed, user_id=2c9c1fff948941b8baacac536fc09724, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:22.549407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:22.565980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 816/1612] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:22 2022] PUT /identity/v3/projects/888be6c04e3e4248a000a1c601a1c8ed/users/2c9c1fff948941b8baacac536fc09724/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:22.567073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:22.660451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2520b3ef-cb29-45a9-b695-bd3e35fc0fa7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:22.704298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f75d58-71d3-4b6e-b165-d0891c724e38 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:22.714296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:22.714963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.715248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.715525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.762016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:22.804381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257f872d-e068-4aff-b9f1-a9e6135acf06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:22.808064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257f872d-e068-4aff-b9f1-a9e6135acf06 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.808506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257f872d-e068-4aff-b9f1-a9e6135acf06 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.808900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257f872d-e068-4aff-b9f1-a9e6135acf06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.837456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:22.896645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:22.902646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:22.909479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257f872d-e068-4aff-b9f1-a9e6135acf06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:22.911506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257f872d-e068-4aff-b9f1-a9e6135acf06 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:22.928340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 797/1613] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:22.929736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.930076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.930991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.934950 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 817/1614] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:22 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:22.949079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:22.962741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] MFA Rules not processed for user `e68eb1137cc845148ba17a3d5c46f878`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:22.964025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:22.974517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6006a3b6-5f40-4062-b013-ec32f85b6bb6 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:22.987533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a6b83-f555-4844-a04f-f9a312b9a908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:22.988314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a6b83-f555-4844-a04f-f9a312b9a908 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:22.988604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a6b83-f555-4844-a04f-f9a312b9a908 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:22.988915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a6b83-f555-4844-a04f-f9a312b9a908 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:22.990351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a6b83-f555-4844-a04f-f9a312b9a908 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:22.994356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a6b83-f555-4844-a04f-f9a312b9a908 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.049362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 818/1615] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:22 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:23.051144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1291d4d-198c-439c-b04b-9d52b886ccea None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.051690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1291d4d-198c-439c-b04b-9d52b886ccea None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.053147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1291d4d-198c-439c-b04b-9d52b886ccea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.091548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 798/1616] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:22 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:23.097783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.120282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f1291d4d-198c-439c-b04b-9d52b886ccea None None] MFA Rules not processed for user `2c9c1fff948941b8baacac536fc09724`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:23.121056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6b5d85-7a69-41c9-9672-5f0266e1fde2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.122282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f1291d4d-198c-439c-b04b-9d52b886ccea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:23.250765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3176bc12-fd32-46f5-a061-2bda491eafa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.251462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3176bc12-fd32-46f5-a061-2bda491eafa2 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.251749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3176bc12-fd32-46f5-a061-2bda491eafa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.252085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3176bc12-fd32-46f5-a061-2bda491eafa2 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.253715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 819/1617] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:23.255305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade381d3-5123-4af3-b8fd-67d1b01cced2 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.255775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade381d3-5123-4af3-b8fd-67d1b01cced2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.256453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3176bc12-fd32-46f5-a061-2bda491eafa2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:23.257575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade381d3-5123-4af3-b8fd-67d1b01cced2 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.259632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3176bc12-fd32-46f5-a061-2bda491eafa2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.265007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 820/1618] 10.222.0.33 () {54 vars in 962 bytes} [Tue Jul 19 11:04:23 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:04:23.266957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.267397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.267823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.306989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 799/1619] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:23 2022] POST /identity/v3/users => generated 299 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:23.312089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95f8d7d-52c5-4b81-bff4-152dba12f38f None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.312781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95f8d7d-52c5-4b81-bff4-152dba12f38f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.313212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95f8d7d-52c5-4b81-bff4-152dba12f38f None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.315885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 800/1620] 10.222.0.33 () {54 vars in 962 bytes} [Tue Jul 19 11:04:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:04:23.316890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95f8d7d-52c5-4b81-bff4-152dba12f38f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.349239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95f8d7d-52c5-4b81-bff4-152dba12f38f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.349661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:23.351192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:23.364234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a6395c-9e04-43ef-a4b9-23412031a578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.365266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a6395c-9e04-43ef-a4b9-23412031a578 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.365895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a6395c-9e04-43ef-a4b9-23412031a578 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.366993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a6395c-9e04-43ef-a4b9-23412031a578 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.367951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a6395c-9e04-43ef-a4b9-23412031a578 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:23.370625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a6395c-9e04-43ef-a4b9-23412031a578 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.412459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 801/1621] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:23 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:23.412459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.412459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.412459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.458624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:23.462916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:23.478045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:23.499835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 821/1622] 10.222.0.33 () {58 vars in 1059 bytes} [Tue Jul 19 11:04:23 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:23.501085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.514436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:23.522746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41e984e-9dad-4c9f-b043-ff77b27bd3e7 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.535304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.536218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.536481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.536778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] PATH_INFO: `/v3/projects/af504d9b679b4634a705d1501c3ea9ab/users/f4b0bb81e6734450bd9a55ab4ddbf449/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.569307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af504d9b679b4634a705d1501c3ea9ab, user_id=f4b0bb81e6734450bd9a55ab4ddbf449, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:23.572604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.604774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 822/1623] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:23 2022] PUT /identity/v3/projects/af504d9b679b4634a705d1501c3ea9ab/users/f4b0bb81e6734450bd9a55ab4ddbf449/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:23.605661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.649342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 802/1624] 10.222.0.33 () {58 vars in 1059 bytes} [Tue Jul 19 11:04:23 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:23.650041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.681611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43bd19f-9895-4b56-9f84-f77273f8eaad None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.755527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa9063b-200d-4b6b-8995-c485b9410315 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.763911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.764659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.765230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.765590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.831724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.833638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.835101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.835101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.835943 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:23.841879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.869959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 803/1625] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:23 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:23.874882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.876382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:23.878347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.891111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 823/1626] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:23 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:23.891922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:23.904580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7193546-1d91-482f-9d88-ffc0b7382f83 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.905444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f409cd5-7e78-4720-b148-79c82cbe5f26 service nova] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:23.911794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335f8e0-cecd-4b8b-ac4c-241971429044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.912614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335f8e0-cecd-4b8b-ac4c-241971429044 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.914048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335f8e0-cecd-4b8b-ac4c-241971429044 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.914308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335f8e0-cecd-4b8b-ac4c-241971429044 admin admin] PATH_INFO: `/v3/projects/af504d9b679b4634a705d1501c3ea9ab/users/f4b0bb81e6734450bd9a55ab4ddbf449/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.927887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0335f8e0-cecd-4b8b-ac4c-241971429044 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af504d9b679b4634a705d1501c3ea9ab, user_id=f4b0bb81e6734450bd9a55ab4ddbf449, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:23.931952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0335f8e0-cecd-4b8b-ac4c-241971429044 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:23.934508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:23.935223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.935663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.936001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:23.978377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 824/1627] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:23 2022] PUT /identity/v3/projects/af504d9b679b4634a705d1501c3ea9ab/users/f4b0bb81e6734450bd9a55ab4ddbf449/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:23.983437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45ba821-6515-45fb-8e76-70f511dcce5a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:23.983848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45ba821-6515-45fb-8e76-70f511dcce5a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:23.984436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45ba821-6515-45fb-8e76-70f511dcce5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.061776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a45ba821-6515-45fb-8e76-70f511dcce5a None None] MFA Rules not processed for user `f4b0bb81e6734450bd9a55ab4ddbf449`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:24.064874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a45ba821-6515-45fb-8e76-70f511dcce5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:24.068120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:24.069819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:24.093856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 804/1628] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:24.238903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:24.295237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 825/1629] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:23 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:24.311450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:24.404417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:24.424604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a37c72-418c-46de-9706-284a9600e95d service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:24.456055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1b363-4c53-456a-bcf9-ac6f29cbded8 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.456346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1b363-4c53-456a-bcf9-ac6f29cbded8 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.456684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1b363-4c53-456a-bcf9-ac6f29cbded8 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.458446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 826/1630] 10.222.0.33 () {54 vars in 912 bytes} [Tue Jul 19 11:04:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 11:04:24.466288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.466596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.466879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.517582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:24.518867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:24.536713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f26aee-797a-45d3-aef0-f568276063e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:24.537684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f26aee-797a-45d3-aef0-f568276063e4 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.538690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f26aee-797a-45d3-aef0-f568276063e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.539117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f26aee-797a-45d3-aef0-f568276063e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.564516 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f26aee-797a-45d3-aef0-f568276063e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:24.566068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f26aee-797a-45d3-aef0-f568276063e4 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:24.588840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 805/1631] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:24 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:24.590261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960f0a40-2e4c-452c-8db2-da19b2ef208d None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.590658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960f0a40-2e4c-452c-8db2-da19b2ef208d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.590876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960f0a40-2e4c-452c-8db2-da19b2ef208d None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.592804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 806/1632] 10.222.0.33 () {54 vars in 912 bytes} [Tue Jul 19 11:04:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:04:24.602747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.603103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.603304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.654685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:24.664561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:24.688847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 827/1633] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:04:24 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:24.690929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:24.811845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8874d6f7-31b3-467d-b5be-d76abbd5e761 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:24.822124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:24.823680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.824458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.825014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] PATH_INFO: `/v3/users/bd202e0ca919481583fb2559be68eee0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.827207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd202e0ca919481583fb2559be68eee0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:24.829556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:24.834533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 807/1634] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:04:24 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:04:24.836107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:24.866312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64211231-0cc0-4491-aa5f-ae729d5ab31b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:24.874453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:24.875388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.875883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.876935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.900956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:24.905616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:24.918321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd202e0ca919481583fb2559be68eee0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:24.930649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd202e0ca919481583fb2559be68eee0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:24.940638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd202e0ca919481583fb2559be68eee0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:24.941908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd202e0ca919481583fb2559be68eee0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:24.942832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9359e720-959b-40ef-87e2-8ca4c4c05503 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd202e0ca919481583fb2559be68eee0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:24.947014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 808/1635] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:24 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:24.947716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:24.968175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd205c3b-8379-4e55-8234-22f0874d2159 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:24.981169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:24.985233 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:24.985574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:24.985859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:24.987196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:24.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:25.003726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 828/1636] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:24 2022] DELETE /identity/v3/users/bd202e0ca919481583fb2559be68eee0 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:25.008193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:25.008497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:25.008788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:25.028433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:25.034868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:25.049291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:25.051143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:25.062348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 809/1637] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:24 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:25.273752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 829/1638] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:25.344935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:25.347103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:25.586937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb086529-4545-4615-9d64-9ffee4ddc4b5 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:25.651813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:25.765801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:25.766915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:25.767592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:25.768169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:25.790500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:25.792198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:25.809730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 810/1639] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:28.327015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:28.328734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:28.355147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf36cdc-291a-4b1e-89af-efaa7c5deab4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:28.361088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:28.363399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:28.363691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:28.363967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] PATH_INFO: `/v3/projects/479a805c7ec24812ba05c59b6defa408` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:28.366198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] RBAC: Authorizing `identity:delete_project(project_id=479a805c7ec24812ba05c59b6defa408)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:28.370661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: WARNING py.warnings [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:04:28.370661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: warnings.warn(msg) Jul 19 11:04:28.370661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: Jul 19 11:04:28.374619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:28.525186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '479a805c7ec24812ba05c59b6defa408', 'request_id': 'req-88fc0d95-f37c-436e-824b-4eb566663691'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.534055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '479a805c7ec24812ba05c59b6defa408', 'request_id': 'req-88fc0d95-f37c-436e-824b-4eb566663691'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.555690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '479a805c7ec24812ba05c59b6defa408', 'request_id': 'req-88fc0d95-f37c-436e-824b-4eb566663691'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.606115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] The token cache is being invalidate because project 479a805c7ec24812ba05c59b6defa408 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:28.606419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.608262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.610654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 830/1640] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:28 2022] DELETE /identity/v3/projects/479a805c7ec24812ba05c59b6defa408 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:28.614787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:28.629017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f71289-ab8f-417f-afc6-64a5cb3bae67 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:28.781250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:28.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:28.782964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:28.783240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] PATH_INFO: `/v3/users/f4b0bb81e6734450bd9a55ab4ddbf449` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:28.785809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4b0bb81e6734450bd9a55ab4ddbf449)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:28.787139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: WARNING py.warnings [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 19 11:04:28.787139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: warnings.warn(msg) Jul 19 11:04:28.787139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: Jul 19 11:04:28.794628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:28.899808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4b0bb81e6734450bd9a55ab4ddbf449'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.915476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4b0bb81e6734450bd9a55ab4ddbf449'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.958586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4b0bb81e6734450bd9a55ab4ddbf449'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:28.995219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:28.997002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:29.003156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4b0bb81e6734450bd9a55ab4ddbf449'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.008163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-54ab6e46-e9cd-4e0c-8de0-bf79a3452abf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4b0bb81e6734450bd9a55ab4ddbf449'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.025895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fc0d95-f37c-436e-824b-4eb566663691 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:29.030026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 811/1641] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:28 2022] DELETE /identity/v3/users/f4b0bb81e6734450bd9a55ab4ddbf449 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:29.044238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:29.045520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:29.045810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:29.046073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] PATH_INFO: `/v3/users/e68eb1137cc845148ba17a3d5c46f878` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:29.053427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e68eb1137cc845148ba17a3d5c46f878)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:29.056099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:29.187164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e68eb1137cc845148ba17a3d5c46f878'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.207279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e68eb1137cc845148ba17a3d5c46f878'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.219364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e68eb1137cc845148ba17a3d5c46f878'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.220533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e68eb1137cc845148ba17a3d5c46f878'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.221282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e68eb1137cc845148ba17a3d5c46f878'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:29.233564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:29.252450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 831/1642] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:28 2022] DELETE /identity/v3/users/e68eb1137cc845148ba17a3d5c46f878 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:04:29.257352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:29.257645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:29.257924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:29.295614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:29.297243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:29.416403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:29.466131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 812/1643] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:29.862016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:29.864085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:29.983852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86422d6b-87ca-4261-8c81-83b32e5a02f5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:29.990539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183bbfb1-a768-4696-86a9-17c16e11a58a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:29.992419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183bbfb1-a768-4696-86a9-17c16e11a58a service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:29.992871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183bbfb1-a768-4696-86a9-17c16e11a58a service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:29.993296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183bbfb1-a768-4696-86a9-17c16e11a58a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:30.008964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183bbfb1-a768-4696-86a9-17c16e11a58a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:30.013426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183bbfb1-a768-4696-86a9-17c16e11a58a service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:30.034018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 832/1644] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:30.623423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:30.631133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:30.683953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c05205-a1ce-4843-8b85-65cbc628fe5e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:30.704880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:30.706265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:30.706914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:30.707436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] PATH_INFO: `/v3/projects/af504d9b679b4634a705d1501c3ea9ab` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:30.711772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af504d9b679b4634a705d1501c3ea9ab)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:30.714529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:30.837249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af504d9b679b4634a705d1501c3ea9ab', 'request_id': 'req-4035bcc3-aef2-4826-a2fa-0deef6d09d70'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:30.861949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af504d9b679b4634a705d1501c3ea9ab', 'request_id': 'req-4035bcc3-aef2-4826-a2fa-0deef6d09d70'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:30.877500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af504d9b679b4634a705d1501c3ea9ab', 'request_id': 'req-4035bcc3-aef2-4826-a2fa-0deef6d09d70'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:30.899586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] The token cache is being invalidate because project af504d9b679b4634a705d1501c3ea9ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:30.900728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:30.904176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4035bcc3-aef2-4826-a2fa-0deef6d09d70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:30.906732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 813/1645] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:30 2022] DELETE /identity/v3/projects/af504d9b679b4634a705d1501c3ea9ab => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:30.912075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:30.912559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:30.912628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:30.947521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:30.948987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:31.136244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 833/1646] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:31.143094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.143641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.144087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.167312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:31.168946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:31.190265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 814/1647] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:31.194863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:31.210554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb71d8ab-fb9e-4c83-87f8-e566620c0edc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:31.218985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:31.220342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.221344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.221798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.222806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:31.223287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:31.225951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:31.254313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 834/1648] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:31.255550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:31.271334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3617cdd6-d63f-4c24-a8ff-1dabd4bf50a9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:31.276905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:31.277782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.277869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.278291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.278786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:31.280953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:31.341231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 815/1649] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:31 2022] POST /identity/v3/projects => generated 381 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:31.346948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:31.394409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d97e142-3c32-49d1-ac3c-520558300e5c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:31.577184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:31.578866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.579716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.580947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.582491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:31.588375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:31.682105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:31.683028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 835/1650] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:31 2022] POST /identity/v3/users => generated 291 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:31.713024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8bc364-9008-4622-9553-654995562312 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:31.744047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:31.744932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.758381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.759051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.760666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:31.762572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:31.798538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 816/1651] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:31 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:31.802598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:31.819631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5d91a-69b4-4c38-99a8-3d6871ed2282 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:31.831687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29108e21-4aa3-4508-8322-bbcd71091f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:31.832603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29108e21-4aa3-4508-8322-bbcd71091f61 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.833046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29108e21-4aa3-4508-8322-bbcd71091f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.833464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29108e21-4aa3-4508-8322-bbcd71091f61 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.834394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29108e21-4aa3-4508-8322-bbcd71091f61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:31.839686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29108e21-4aa3-4508-8322-bbcd71091f61 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:31.849513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:31.865744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 836/1652] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:31 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:31.868501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:31.886092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98aef61b-2a16-47bc-98ac-7d194565285a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:31.900898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:31.902358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.902914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.903367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] PATH_INFO: `/v3/projects/d0151c8128cf494284a991baaf7ee36c/users/63788f1e30454ca6915aad7f1ead0920/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:31.946857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0151c8128cf494284a991baaf7ee36c, user_id=63788f1e30454ca6915aad7f1ead0920, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:31.949965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:31.984455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 817/1653] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:31 2022] PUT /identity/v3/projects/d0151c8128cf494284a991baaf7ee36c/users/63788f1e30454ca6915aad7f1ead0920/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:31.996247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:31.997501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:31.998016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:32.054390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] MFA Rules not processed for user `63788f1e30454ca6915aad7f1ead0920`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:32.058062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:32.109271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:32.112461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:32.293428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 837/1654] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:32.298121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:32.352324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62866b90-b559-410d-8876-c1ff48fc0e94 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:32.374525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:32.376169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:32.376590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:32.377165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] PATH_INFO: `/v3/projects/7d0d678908014f8dae55bc6475b84680` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:32.380797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d0d678908014f8dae55bc6475b84680)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:32.384523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:32.408423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:32.483862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559bef20-eb7b-4b18-911f-6cd93687b916 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:32.528229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d0d678908014f8dae55bc6475b84680', 'request_id': 'req-62912f28-ff14-4393-b8d1-b654f22c47e2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:32.552455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d0d678908014f8dae55bc6475b84680', 'request_id': 'req-62912f28-ff14-4393-b8d1-b654f22c47e2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:32.568676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d0d678908014f8dae55bc6475b84680', 'request_id': 'req-62912f28-ff14-4393-b8d1-b654f22c47e2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:32.587205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] The token cache is being invalidate because project 7d0d678908014f8dae55bc6475b84680 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:32.587600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:32.588447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-62912f28-ff14-4393-b8d1-b654f22c47e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:32.596004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 818/1655] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:32 2022] DELETE /identity/v3/projects/7d0d678908014f8dae55bc6475b84680 => generated 0 bytes in 483 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:32.598644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a2db6b-2b6a-4a29-8a7b-467abbae1e89 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:32.599031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a2db6b-2b6a-4a29-8a7b-467abbae1e89 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:32.599607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a2db6b-2b6a-4a29-8a7b-467abbae1e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:32.633740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:32.635335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:32.635335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:32.635335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:32.636571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:32.638989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:32.667482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-72a2db6b-2b6a-4a29-8a7b-467abbae1e89 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:32.671826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-72a2db6b-2b6a-4a29-8a7b-467abbae1e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:32.705483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 838/1656] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:32 2022] POST /identity/v3/projects => generated 379 bytes in 403 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:32.705483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:32.708066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:32.709113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 819/1657] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:32.713756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:32.714115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:32.714464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:32.762587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:32.776458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:32.799394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:32.970971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53b40ed-a5b4-4704-b40e-e3d95df0ef55 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:32.988317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:32.989121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:32.991935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:32.992264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.010196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.017454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.017742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 820/1658] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:33.023904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.049214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52de090d-92ce-4637-b9e2-cb7310488407 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.090731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 839/1659] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:32 2022] POST /identity/v3/users => generated 296 bytes in 386 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:33.091743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.092438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.093045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.093561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.094438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:33.095004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.096771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.097107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.142413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 821/1660] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:33.143658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.153449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa1d6f-8433-44bd-8b52-55ef39ea9fbd admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.169279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed94ad-5c89-47d6-87aa-6b94107b27f5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.173061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.173997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.174287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.174562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.175301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.177204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.186588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.188590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.189015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.190328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.191174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.198365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.201408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 840/1661] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:33.206244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.227633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906e5b8d-f8aa-42a5-b4c5-e12764eda5e4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.247286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e052d0-39eb-4955-b749-a73e8b1ebedd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.248452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e052d0-39eb-4955-b749-a73e8b1ebedd admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.248905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e052d0-39eb-4955-b749-a73e8b1ebedd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.249306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e052d0-39eb-4955-b749-a73e8b1ebedd admin admin] PATH_INFO: `/v3/projects/de67cc0666ca41b68958a407a0f989c9/users/241a9fa6d1c14d40a92078af2294172d/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.273775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 822/1662] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:33 2022] POST /identity/v3/projects => generated 363 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:33.274994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.283667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e052d0-39eb-4955-b749-a73e8b1ebedd admin admin] RBAC: Authorizing `identity:create_grant(project_id=de67cc0666ca41b68958a407a0f989c9, user_id=241a9fa6d1c14d40a92078af2294172d, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.286826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e052d0-39eb-4955-b749-a73e8b1ebedd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.336980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 841/1663] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:33 2022] PUT /identity/v3/projects/de67cc0666ca41b68958a407a0f989c9/users/241a9fa6d1c14d40a92078af2294172d/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:33.339357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.339357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.340000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.405535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] MFA Rules not processed for user `f6cb55e4c41f4e6d94156c597a1baad8`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:33.412105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:33.529419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:33.552553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e82bb2-dcb7-4691-b4fa-5b921d302fa4 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.627293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 842/1664] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:33.628505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.750152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d52306-0a21-48ef-99d3-b59c1f2b15ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.751562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d52306-0a21-48ef-99d3-b59c1f2b15ef admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.752094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d52306-0a21-48ef-99d3-b59c1f2b15ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.752627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d52306-0a21-48ef-99d3-b59c1f2b15ef admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.753706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d52306-0a21-48ef-99d3-b59c1f2b15ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.758032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d52306-0a21-48ef-99d3-b59c1f2b15ef admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.772801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfe4f15-209e-4b44-a766-164ab23a6168 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.786478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.788726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.789518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 823/1665] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:33 2022] POST /identity/v3/users => generated 282 bytes in 515 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:33.791546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.792133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.794199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.794527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.794907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.795702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.799373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.816146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] MFA Rules not processed for user `f6cb55e4c41f4e6d94156c597a1baad8`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:33.817820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:33.822434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 843/1666] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:33.823559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.834720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 824/1667] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:33.836255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.840623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:33.851775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de92a96-bf14-4e43-bb18-b290983e1781 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.854196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e6aa4c-c2d8-4f63-9433-f3fe3cc1f51e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:33.860446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.862709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.863025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.863496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] PATH_INFO: `/v3/projects/de67cc0666ca41b68958a407a0f989c9/users/241a9fa6d1c14d40a92078af2294172d/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.864636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:33.865351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:33.865642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:33.865909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:33.866981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.867087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de67cc0666ca41b68958a407a0f989c9, user_id=241a9fa6d1c14d40a92078af2294172d, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:33.868306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.869819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:33.897908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 844/1669] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:33.898911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.902212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 825/1669] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:33 2022] PUT /identity/v3/projects/de67cc0666ca41b68958a407a0f989c9/users/241a9fa6d1c14d40a92078af2294172d/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:33.903265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:33.905336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:34.005353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76403412-8236-40e8-8b46-d9b032715406 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:34.012326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fb8469-271d-4c91-8066-87a943a2791a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:34.012326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fb8469-271d-4c91-8066-87a943a2791a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.012571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fb8469-271d-4c91-8066-87a943a2791a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.012872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fb8469-271d-4c91-8066-87a943a2791a admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.013829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fb8469-271d-4c91-8066-87a943a2791a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:34.016765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fb8469-271d-4c91-8066-87a943a2791a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:34.031072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 826/1670] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:34.032419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.032918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.033019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.064144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee16e9-79ae-4d1d-84ec-3de344bf3bc8 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:34.073118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] MFA Rules not processed for user `241a9fa6d1c14d40a92078af2294172d`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:34.074880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:34.204806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:34.225578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:34.226247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.226671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.226957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.238443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 827/1671] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:34 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:34.241052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:34.274421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c98b6-e81a-4cc1-b95a-3c73f65c0bc7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:34.291121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:34.293099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.293354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.293695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] PATH_INFO: `/v3/projects/40227f72f67c4d4d93515bf58b3a697e/users/1c12bf5404e84ca892cc686b34d4521e/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.339043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40227f72f67c4d4d93515bf58b3a697e, user_id=1c12bf5404e84ca892cc686b34d4521e, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:34.349793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:34.380543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 828/1672] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:34 2022] PUT /identity/v3/projects/40227f72f67c4d4d93515bf58b3a697e/users/1c12bf5404e84ca892cc686b34d4521e/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:34.391756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:34.392433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:34.394357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:34.395768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:34.408192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 845/1673] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:33 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:34.409083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:34.410426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:34.438507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c1a4a9-a4e5-4277-8503-78e730c727d4 service nova] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:34.493860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:34.563531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce74a05-d16e-4693-aa11-2e673d19ae65 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:34.568963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ac8e6-e724-4bdb-a102-f4dd425b8092 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:34.569896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ac8e6-e724-4bdb-a102-f4dd425b8092 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.570310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ac8e6-e724-4bdb-a102-f4dd425b8092 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.570521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:34.570796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ac8e6-e724-4bdb-a102-f4dd425b8092 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.572586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.573377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.573657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] PATH_INFO: `/v3/users/2c9c1fff948941b8baacac536fc09724` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.575171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c9c1fff948941b8baacac536fc09724)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:34.577413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:34.695878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c9c1fff948941b8baacac536fc09724'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:34.709017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c9c1fff948941b8baacac536fc09724'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:34.720364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990ac8e6-e724-4bdb-a102-f4dd425b8092 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:34.723173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990ac8e6-e724-4bdb-a102-f4dd425b8092 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:34.726408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c9c1fff948941b8baacac536fc09724'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:34.728257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c9c1fff948941b8baacac536fc09724'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:34.729039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f0dc80ef-8bcc-428d-b1c9-40805babd9b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c9c1fff948941b8baacac536fc09724'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:34.736882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 829/1674] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:34 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:34.738348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c861f0b6-560b-4773-849d-a571d04e4055 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.738746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c861f0b6-560b-4773-849d-a571d04e4055 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.739121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c861f0b6-560b-4773-849d-a571d04e4055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.796202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 846/1675] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:34 2022] DELETE /identity/v3/users/2c9c1fff948941b8baacac536fc09724 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:34.799586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:34.799893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:34.800174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:34.814664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c861f0b6-560b-4773-849d-a571d04e4055 None None] MFA Rules not processed for user `1c12bf5404e84ca892cc686b34d4521e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:34.816973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c861f0b6-560b-4773-849d-a571d04e4055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:34.895960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:34.908148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.004324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 830/1676] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:34 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:35.011700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.012019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.012353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.055242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:35.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] MFA Rules not processed for user `c95281b28eae450d8678583a929c3b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:35.082355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.082492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 847/1677] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:35.082492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.082951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.113033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0463711-1f2b-4f57-a088-22c8141ea0d2 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:35.128241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:35.128999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.129226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.129500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.130200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:35.132149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:35.186542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 848/1678] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:35 2022] POST /identity/v3/projects => generated 361 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:35.190734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.196687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.243214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 831/1679] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:35.243214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.266128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe818445-0014-44d9-a959-a5ea27003247 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:35.271713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:35.319822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac845eac-fc48-4558-8200-d840bd53d3bf admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:35.327448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51d6c7c-76db-4d6d-a0b7-5aa74b6b26ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:35.328448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51d6c7c-76db-4d6d-a0b7-5aa74b6b26ed service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.328763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51d6c7c-76db-4d6d-a0b7-5aa74b6b26ed service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.329185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51d6c7c-76db-4d6d-a0b7-5aa74b6b26ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.503594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51d6c7c-76db-4d6d-a0b7-5aa74b6b26ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:35.507340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51d6c7c-76db-4d6d-a0b7-5aa74b6b26ed service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:35.536662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb509a32-e172-4942-9c00-246ab03abd7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:35.537502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb509a32-e172-4942-9c00-246ab03abd7b admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.539557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb509a32-e172-4942-9c00-246ab03abd7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.539693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb509a32-e172-4942-9c00-246ab03abd7b admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.540510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb509a32-e172-4942-9c00-246ab03abd7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:35.545130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb509a32-e172-4942-9c00-246ab03abd7b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:35.545665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 849/1680] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:35.549364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.550711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.551172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.590367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 832/1681] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:35 2022] POST /identity/v3/users => generated 287 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:35.592015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b460c2-dc72-470c-91ac-5df19cbdac0f None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.592493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b460c2-dc72-470c-91ac-5df19cbdac0f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.592883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b460c2-dc72-470c-91ac-5df19cbdac0f None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.594591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 833/1682] 10.222.0.33 () {54 vars in 912 bytes} [Tue Jul 19 11:04:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:04:35.595469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b460c2-dc72-470c-91ac-5df19cbdac0f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.599206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a0b460c2-dc72-470c-91ac-5df19cbdac0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.611996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] MFA Rules not processed for user `c95281b28eae450d8678583a929c3b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:35.615805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.733991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 850/1683] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:35.733991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.748074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f951f-6118-4827-8eb2-201c3080ca8a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:35.748678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b460c2-dc72-470c-91ac-5df19cbdac0f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:35.758582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c2e97f-9c3d-46d4-a474-31b4135c661d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:35.759625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c2e97f-9c3d-46d4-a474-31b4135c661d admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.760097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c2e97f-9c3d-46d4-a474-31b4135c661d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.760743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c2e97f-9c3d-46d4-a474-31b4135c661d admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.761924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c2e97f-9c3d-46d4-a474-31b4135c661d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:35.763332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '241a9fa6d1c14d40a92078af2294172d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'de67cc0666ca41b68958a407a0f989c9', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:35.764441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c2e97f-9c3d-46d4-a474-31b4135c661d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:35.767197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.767558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.767868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.769651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:35.769981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:35.779056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 851/1684] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:35 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:35.779599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:35.781110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.781409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.781741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.831336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 834/1685] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:35.852406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.857422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.859722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:35.865239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:35.901221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:35.906433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 852/1686] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:04:35 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:35.907504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:35.969445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a918f192-b9c8-44d8-99f2-2f31e76d31d7 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:35.982610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:35.983859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:35.984369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:35.984997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] PATH_INFO: `/v3/projects/2ee71720233040ccbc5b7d058c832dd1/users/c86e6df734d14b7c8707da98b1414d78/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:35.993481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb842764-88e1-4604-9639-d5595b049ced tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.010052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.011960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.013005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.013704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.016937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ee71720233040ccbc5b7d058c832dd1, user_id=c86e6df734d14b7c8707da98b1414d78, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.022529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.030870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.033493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.047898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 853/1687] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:35 2022] PUT /identity/v3/projects/2ee71720233040ccbc5b7d058c832dd1/users/c86e6df734d14b7c8707da98b1414d78/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:36.049651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.068728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 835/1688] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:35 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:36.072199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.105507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ee71f-3d8c-48d0-8f4f-1922d0b09dd5 service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.148859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:36.220173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1418d6bc-9d53-4540-808e-9d527cb26b01 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.231628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '241a9fa6d1c14d40a92078af2294172d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'de67cc0666ca41b68958a407a0f989c9', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.233882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.233882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.235488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.236387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.238782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.242157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.243050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.243403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.243786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.244633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.248552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.275588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 836/1689] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:36 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:36.278106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.281084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:36.296299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 854/1690] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:36 2022] POST /identity/v3/projects => generated 353 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:36.297557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.373176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5672d9df-4afb-4a4b-9191-d5a5842b1974 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.380172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8976658-ed7f-46fb-ab5c-0efef588dffa tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.386734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.388061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.389407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.390063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] PATH_INFO: `/v3/projects/2ee71720233040ccbc5b7d058c832dd1/users/c86e6df734d14b7c8707da98b1414d78/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.392811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ee71720233040ccbc5b7d058c832dd1, user_id=c86e6df734d14b7c8707da98b1414d78, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.396483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.421114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 855/1691] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:36 2022] PUT /identity/v3/projects/2ee71720233040ccbc5b7d058c832dd1/users/c86e6df734d14b7c8707da98b1414d78/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:36.422403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.442209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44093562-8605-47f2-84ba-100baac151c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.442858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44093562-8605-47f2-84ba-100baac151c1 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.443154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44093562-8605-47f2-84ba-100baac151c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.443415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44093562-8605-47f2-84ba-100baac151c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.453655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44093562-8605-47f2-84ba-100baac151c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.455104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44093562-8605-47f2-84ba-100baac151c1 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.469294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 837/1692] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:36 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:36.470710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.471064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.471344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.479714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:36.510150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] MFA Rules not processed for user `c86e6df734d14b7c8707da98b1414d78`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:36.511660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:36.523095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3457bcc-f7a0-4f80-b25b-d55c40536a2b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.535378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '241a9fa6d1c14d40a92078af2294172d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'de67cc0666ca41b68958a407a0f989c9', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.537450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.538135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.538646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.542827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:36.543564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.547262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.578293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 856/1693] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:36.654433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 838/1694] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:36 2022] POST /identity/v3/auth/tokens => generated 3295 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:36.667207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.673201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:36.674994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:36.683552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a849fc1d-b963-485b-bfb1-cd3a4beede53 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.702747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '241a9fa6d1c14d40a92078af2294172d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'de67cc0666ca41b68958a407a0f989c9', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.707464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.709780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.712287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] PATH_INFO: `/v3/projects/1be0aac0cecb4a3793ef1ecb94bc2c74` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.722898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorizing `identity:delete_project(project_id=1be0aac0cecb4a3793ef1ecb94bc2c74)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:36.728603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:36.824599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:36.850740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75852a09-ccf5-4116-98e4-58aad0a7b4e8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:36.868531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1be0aac0cecb4a3793ef1ecb94bc2c74', 'request_id': 'req-e806813d-b545-418c-ab3b-ac7e10d6a627'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:36.885212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1be0aac0cecb4a3793ef1ecb94bc2c74', 'request_id': 'req-e806813d-b545-418c-ab3b-ac7e10d6a627'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:36.901496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1be0aac0cecb4a3793ef1ecb94bc2c74', 'request_id': 'req-e806813d-b545-418c-ab3b-ac7e10d6a627'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:36.960144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] The token cache is being invalidate because project 1be0aac0cecb4a3793ef1ecb94bc2c74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:36.960674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:36.961523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:36.963869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 857/1695] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:36 2022] DELETE /identity/v3/projects/1be0aac0cecb4a3793ef1ecb94bc2c74 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:36.993777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:36.994493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:36.994796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:36.995109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:36.998110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:37.133728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:37.156574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:37.159272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61726d3-e064-4f80-a07f-008f5f1c2b37 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:37.192108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 839/1696] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:36 2022] GET /identity/v3/auth/tokens => generated 3295 bytes in 519 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:37.446343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:37.448513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:37.692993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806813d-b545-418c-ab3b-ac7e10d6a627 tempest-VolumeQuotaClassesTest-980762111 tempest-VolumeQuotaClassesTest-980762111-project-admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:37.709350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:37.712181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:37.712181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:37.712885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] PATH_INFO: `/v3/projects/888be6c04e3e4248a000a1c601a1c8ed` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:37.718520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=888be6c04e3e4248a000a1c601a1c8ed)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:37.723219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:37.734425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6c346c-a223-4c42-b2a6-ae153800812b None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:37.735183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6c346c-a223-4c42-b2a6-ae153800812b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:37.735826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6c346c-a223-4c42-b2a6-ae153800812b None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:37.738115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 840/1697] 10.222.0.33 () {54 vars in 912 bytes} [Tue Jul 19 11:04:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 11:04:37.739383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6c346c-a223-4c42-b2a6-ae153800812b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:37.743063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ce6c346c-a223-4c42-b2a6-ae153800812b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:37.811517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '888be6c04e3e4248a000a1c601a1c8ed', 'request_id': 'req-625de7da-4c9f-4f31-9ac6-2f4197412f5b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:37.818921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6c346c-a223-4c42-b2a6-ae153800812b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:37.820635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '888be6c04e3e4248a000a1c601a1c8ed', 'request_id': 'req-625de7da-4c9f-4f31-9ac6-2f4197412f5b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:37.845197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '888be6c04e3e4248a000a1c601a1c8ed', 'request_id': 'req-625de7da-4c9f-4f31-9ac6-2f4197412f5b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:37.871149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] The token cache is being invalidate because project 888be6c04e3e4248a000a1c601a1c8ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:37.872000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:37.872853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-625de7da-4c9f-4f31-9ac6-2f4197412f5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:37.874932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 858/1698] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:37 2022] DELETE /identity/v3/projects/888be6c04e3e4248a000a1c601a1c8ed => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:37.882358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae88595-fd7f-4098-b03d-deb9b123a75f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:37.883407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae88595-fd7f-4098-b03d-deb9b123a75f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:37.884058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae88595-fd7f-4098-b03d-deb9b123a75f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:37.937249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:37.937716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8ae88595-fd7f-4098-b03d-deb9b123a75f None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:37.938687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8ae88595-fd7f-4098-b03d-deb9b123a75f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:37.939390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:37.939881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:37.940462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] PATH_INFO: `/v3/users/63788f1e30454ca6915aad7f1ead0920` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:37.943376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] RBAC: Authorizing `identity:delete_user(user_id=63788f1e30454ca6915aad7f1ead0920)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:37.946319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:38.046313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63788f1e30454ca6915aad7f1ead0920'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.071918 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63788f1e30454ca6915aad7f1ead0920'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.105057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63788f1e30454ca6915aad7f1ead0920'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.122435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:38.140518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 859/1699] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:04:37 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:38.142843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72deccbd-bbd1-4659-9f04-7ff630e5c083 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.143571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72deccbd-bbd1-4659-9f04-7ff630e5c083 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.143965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72deccbd-bbd1-4659-9f04-7ff630e5c083 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:38.145614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 860/1700] 10.222.0.33 () {54 vars in 961 bytes} [Tue Jul 19 11:04:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:04:38.146771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.147161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.147530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:38.192492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:38.201034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:38.205324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63788f1e30454ca6915aad7f1ead0920'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.211664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-227250eb-e35a-4576-9e6f-2c0da64ce81b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63788f1e30454ca6915aad7f1ead0920'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.227528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 861/1701] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:38.228495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:38.247760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 841/1702] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:37 2022] DELETE /identity/v3/users/63788f1e30454ca6915aad7f1ead0920 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:38.260819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:38.266094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea9f229-3773-469c-8e0f-1ddc43655f4c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.266395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea9f229-3773-469c-8e0f-1ddc43655f4c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.266677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea9f229-3773-469c-8e0f-1ddc43655f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:38.307058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3654c4-446a-49bf-a37f-3d9a51709bdb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:38.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-dea9f229-3773-469c-8e0f-1ddc43655f4c None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:38.368382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-dea9f229-3773-469c-8e0f-1ddc43655f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:38.530038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:38.533882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.534655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.537988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] PATH_INFO: `/v3/users/d6d30fa2b0274f7087c8ba3394bafb44` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:38.541531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6d30fa2b0274f7087c8ba3394bafb44)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:38.543590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:38.617294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 842/1703] 10.222.0.33 () {58 vars in 1058 bytes} [Tue Jul 19 11:04:38 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 369 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:38.618442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.618727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.619000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:38.684686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:38.686486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:38.696368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6d30fa2b0274f7087c8ba3394bafb44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.707849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 843/1704] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:38.707849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:38.710166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:38.717565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6d30fa2b0274f7087c8ba3394bafb44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.734237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:38.752596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6d30fa2b0274f7087c8ba3394bafb44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.757203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6d30fa2b0274f7087c8ba3394bafb44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.758468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-127134f8-6948-4c24-8283-1a6a417ce649 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6d30fa2b0274f7087c8ba3394bafb44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:38.844812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 862/1705] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:38 2022] DELETE /identity/v3/users/d6d30fa2b0274f7087c8ba3394bafb44 => generated 0 bytes in 614 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:38.846573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.846868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.848409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:38.889920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a22030a-a2fc-48da-938d-254c6224696d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:38.909807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:38.911044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:38.996167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:38.997280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:38.997569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:38.997849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.008782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.021353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 863/1706] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:39.022686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.026477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:39.028663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.029173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:39.043145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8457b2a2-47ee-4c90-b24a-01867f8aa0b1 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:39.045101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 844/1707] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:38 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:39.045877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.049412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.054235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:39.055142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.055672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.055672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] PATH_INFO: `/v3/users/1c12bf5404e84ca892cc686b34d4521e` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.060174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c12bf5404e84ca892cc686b34d4521e)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:39.062257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:39.072308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:39.168438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af22ae-a0e3-4417-b078-20e0b8c6b217 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:39.169440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c12bf5404e84ca892cc686b34d4521e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:39.175879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:39.176576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.176895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.177174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.182337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.190476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c12bf5404e84ca892cc686b34d4521e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:39.199459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:39.200088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c12bf5404e84ca892cc686b34d4521e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:39.201908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:39.202867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c12bf5404e84ca892cc686b34d4521e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:39.203911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8ccf8887-15df-4dd8-ae3e-d8e4e48f6e29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c12bf5404e84ca892cc686b34d4521e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:39.222140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 845/1708] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:39 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:39.238295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.244063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.252196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 864/1709] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:39 2022] DELETE /identity/v3/users/1c12bf5404e84ca892cc686b34d4521e => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:39.252196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7159fa1f-95fc-4a0d-acda-2a294f93c8c2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.252196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7159fa1f-95fc-4a0d-acda-2a294f93c8c2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.256807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7159fa1f-95fc-4a0d-acda-2a294f93c8c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.274851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65976a46-c212-49b6-be96-479f2dccdbec service placement] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:39.313444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7159fa1f-95fc-4a0d-acda-2a294f93c8c2 None None] MFA Rules not processed for user `246dc188d9a44b1cb38721e3dd8a863a`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:39.314733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7159fa1f-95fc-4a0d-acda-2a294f93c8c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.418783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:39.451596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:39.451596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.451596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.451895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.453245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:39.453245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:39.456101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcbe5a6-0638-4682-8e5b-132ad30b0175 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:39.482259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 846/1710] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:39.484367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.484661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.484972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.523916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:39.525693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.562171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 865/1711] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:39.567257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 847/1712] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:39.568384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.574405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.578022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.578510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.578915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.652807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] MFA Rules not processed for user `b8f2a78f3a3e4b8e8937e3bc39524690`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:39.655640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.726352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf4cb69-1bd9-4a18-a2b5-c327d1e8ad6c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:39.736074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:39.740524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.740524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.743407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.753949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.769936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:39.773393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec38d851-b1c4-4a67-b3a1-4a2b240e9b91 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:39.813510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 848/1713] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:39.815162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.816815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.819991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.846955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 866/1714] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:39.846955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.846955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.858442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0418f-396e-48dd-bd17-2b9200dbff3d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:39.859376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:39.871893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:39.873661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:39.874123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:39.874511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:39.874641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:39.876129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:39.882835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:39.886015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:39.910610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 849/1715] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:39.925531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 867/1716] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:39 2022] POST /identity/v3/projects => generated 369 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:39.925905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.926789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:39.941969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:40.130383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bac62c-61ae-46de-af82-47acb4d347ac admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.135611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.136549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.137902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.138478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.178387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a85e0d-9a4a-415f-bfdd-5971caa57423 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.295267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.296500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.310186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 868/1717] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:40.311339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:40.313967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:40.354518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1661837f-1b22-4810-bb53-0d0e8f7e4231 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.369772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.370361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.372101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.373594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.374099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.396505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.396908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.397741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.399516 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:40.441533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.446535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.491760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 869/1718] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:40.506069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:40.526792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.528989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.541140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d82c309-3bad-4a93-9bce-46eb6c67d073 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.551810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 850/1719] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 641 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:40.555443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:40.556431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.558343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:40.559065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.561542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.561814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.562555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.583709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.622931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f9d33b-529c-44d3-986e-616d84c10007 service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.645376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6805c5f-21e5-431f-879e-853b4b8dfcfc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.649299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 870/1720] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:40 2022] POST /identity/v3/users => generated 285 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:40.649456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6805c5f-21e5-431f-879e-853b4b8dfcfc service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.649456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6805c5f-21e5-431f-879e-853b4b8dfcfc service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.649456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6805c5f-21e5-431f-879e-853b4b8dfcfc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:40.687036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff06145f-b7c2-40eb-b3bf-34076bd69c65 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.713096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.714462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.714763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.717367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.725459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.727189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.809694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 871/1721] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:40 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:40.823354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:40.905046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fc0e18-b448-42d1-8aa9-44242384e809 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.917970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:40.919015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:40.919486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:40.919940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:40.920898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.922812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.959944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 872/1722] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:40 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:40.961046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:40.965817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6805c5f-21e5-431f-879e-853b4b8dfcfc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:40.968117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6805c5f-21e5-431f-879e-853b4b8dfcfc service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:40.987150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5800b9cb-5b8d-4aa4-85d5-d22dc0c64e03 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:40.999793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d883f9-ec0f-4e6c-a928-e5cff5797e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:41.001171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d883f9-ec0f-4e6c-a928-e5cff5797e9c admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:41.001594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d883f9-ec0f-4e6c-a928-e5cff5797e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:41.001943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d883f9-ec0f-4e6c-a928-e5cff5797e9c admin admin] PATH_INFO: `/v3/projects/a591c3a3a0694cf1a18b4a8b8e1bb2e8/users/9966cbd5be744463b35ee84f5dd655cf/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:41.010394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 851/1723] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:41.015033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:41.016385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:41.017138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:41.055552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d883f9-ec0f-4e6c-a928-e5cff5797e9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a591c3a3a0694cf1a18b4a8b8e1bb2e8, user_id=9966cbd5be744463b35ee84f5dd655cf, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:41.058355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d883f9-ec0f-4e6c-a928-e5cff5797e9c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:41.092058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 873/1724] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:40 2022] PUT /identity/v3/projects/a591c3a3a0694cf1a18b4a8b8e1bb2e8/users/9966cbd5be744463b35ee84f5dd655cf/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:41.107568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:41.107895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:41.108183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:41.125922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] MFA Rules not processed for user `c95281b28eae450d8678583a929c3b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:41.128133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:41.243246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 852/1725] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:41.278626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] MFA Rules not processed for user `9966cbd5be744463b35ee84f5dd655cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:41.279890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:41.298464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:41.394929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:41.396820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:41.569732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:41.600218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fa2c15-a158-4e41-9424-de32723855f9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:41.613188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 874/1726] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:41 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 507 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:41.615658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:41.617645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:41.617937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:41.618214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] PATH_INFO: `/v3/projects/d6f07e7df33a48d7a9820f4100e7caee` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:41.618413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:41.626877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6f07e7df33a48d7a9820f4100e7caee)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:41.633705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:41.673492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:41.693659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a9c9a8-7c11-4f03-8d7a-de4cc552a2de None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:41.721135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:41.722025 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:41.722310 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:41.722582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] PATH_INFO: `/v3/projects/d0151c8128cf494284a991baaf7ee36c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:41.734662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0151c8128cf494284a991baaf7ee36c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:41.747406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:41.904058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0151c8128cf494284a991baaf7ee36c', 'request_id': 'req-6c63cb1e-dee9-4949-b9db-49d99953f14f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.905886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6f07e7df33a48d7a9820f4100e7caee', 'request_id': 'req-2cb8a348-d9a2-47ee-8d43-0765f805dc81'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.916967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6f07e7df33a48d7a9820f4100e7caee', 'request_id': 'req-2cb8a348-d9a2-47ee-8d43-0765f805dc81'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.929831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6f07e7df33a48d7a9820f4100e7caee', 'request_id': 'req-2cb8a348-d9a2-47ee-8d43-0765f805dc81'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.931785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0151c8128cf494284a991baaf7ee36c', 'request_id': 'req-6c63cb1e-dee9-4949-b9db-49d99953f14f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.972516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0151c8128cf494284a991baaf7ee36c', 'request_id': 'req-6c63cb1e-dee9-4949-b9db-49d99953f14f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.975659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] The token cache is being invalidate because project d6f07e7df33a48d7a9820f4100e7caee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:41.976065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.976856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.982332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 853/1727] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:41 2022] DELETE /identity/v3/projects/d6f07e7df33a48d7a9820f4100e7caee => generated 0 bytes in 585 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:41.986757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:41.988483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:41.994414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] The token cache is being invalidate because project d0151c8128cf494284a991baaf7ee36c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:41.994714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:41.995710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.006019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 875/1728] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:41 2022] DELETE /identity/v3/projects/d0151c8128cf494284a991baaf7ee36c => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:42.011156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:42.014181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:42.187014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb8a348-d9a2-47ee-8d43-0765f805dc81 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:42.203325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff92a19d-4a11-43ea-9f6e-8967f04e1673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:42.206061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff92a19d-4a11-43ea-9f6e-8967f04e1673 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:42.206368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff92a19d-4a11-43ea-9f6e-8967f04e1673 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:42.208034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff92a19d-4a11-43ea-9f6e-8967f04e1673 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:42.210086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff92a19d-4a11-43ea-9f6e-8967f04e1673 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:42.215400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c63cb1e-dee9-4949-b9db-49d99953f14f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:42.223834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff92a19d-4a11-43ea-9f6e-8967f04e1673 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:42.233760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:42.234658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:42.234937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:42.235209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] PATH_INFO: `/v3/projects/40227f72f67c4d4d93515bf58b3a697e` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:42.237549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40227f72f67c4d4d93515bf58b3a697e)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:42.239484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:42.321820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 854/1729] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:41 2022] POST /identity/v3/projects => generated 371 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:42.329981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:42.331872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:42.333294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:42.356440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:42.397510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:42.402513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:42.414632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40227f72f67c4d4d93515bf58b3a697e', 'request_id': 'req-499176e8-f26e-418c-a6b5-1a39a0a96772'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.435112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40227f72f67c4d4d93515bf58b3a697e', 'request_id': 'req-499176e8-f26e-418c-a6b5-1a39a0a96772'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.461327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40227f72f67c4d4d93515bf58b3a697e', 'request_id': 'req-499176e8-f26e-418c-a6b5-1a39a0a96772'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.482298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] The token cache is being invalidate because project 40227f72f67c4d4d93515bf58b3a697e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:42.483420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.485612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.494359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 876/1730] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:42 2022] DELETE /identity/v3/projects/40227f72f67c4d4d93515bf58b3a697e => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:42.494359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:42.499605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:42.592066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 855/1731] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:42.594347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:42.604663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:42.641575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeca4599-541a-44c8-8ac5-a0ecc4aa7d47 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:42.647923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:42.648847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:42.649260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:42.649727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:42.650444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:42.656315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:42.667317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:42.693696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 856/1732] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:42 2022] POST /identity/v3/users => generated 292 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:42.695311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:42.697578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:42.721746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dbca03-e4b4-4774-b24b-e575e71788cf admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:42.728550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:42.728932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499176e8-f26e-418c-a6b5-1a39a0a96772 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:42.729561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:42.729846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:42.730721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] PATH_INFO: `/v3/users/c86e6df734d14b7c8707da98b1414d78` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:42.732385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c86e6df734d14b7c8707da98b1414d78)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:42.744488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:42.747535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:42.748524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:42.749143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] PATH_INFO: `/v3/users/241a9fa6d1c14d40a92078af2294172d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:42.749734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:42.753441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=241a9fa6d1c14d40a92078af2294172d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:42.759093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:42.944967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c86e6df734d14b7c8707da98b1414d78'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.964771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c86e6df734d14b7c8707da98b1414d78'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:42.987740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c86e6df734d14b7c8707da98b1414d78'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.054383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '241a9fa6d1c14d40a92078af2294172d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.055410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c86e6df734d14b7c8707da98b1414d78'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.060300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a99f1c28-a0a4-4e3f-93fa-d5434437bf30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c86e6df734d14b7c8707da98b1414d78'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.079255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 857/1733] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:42 2022] DELETE /identity/v3/users/c86e6df734d14b7c8707da98b1414d78 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:43.087133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '241a9fa6d1c14d40a92078af2294172d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.097447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:43.097736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:43.099975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:43.133700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '241a9fa6d1c14d40a92078af2294172d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.135054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '241a9fa6d1c14d40a92078af2294172d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.136892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '241a9fa6d1c14d40a92078af2294172d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:43.204037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:43.205646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:43.207513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 877/1734] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:42 2022] DELETE /identity/v3/users/241a9fa6d1c14d40a92078af2294172d => generated 0 bytes in 715 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:43.210480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:43.229164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:43.409739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:43.465754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 858/1735] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 386 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:43.466711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:43.493559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba8726-527c-432e-b3f6-bc7a053abfd2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:43.502240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de20fa-12ab-4392-a023-2f1e686d92fc admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:43.505017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:43.509197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:43.510134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:43.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:43.511029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:43.512063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:43.512063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:43.512063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:43.513454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:43.515781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:43.516841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:43.517171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:43.527888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:43.570705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 878/1736] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:43 2022] GET /identity/v3/roles => generated 1420 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:43.571833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:43.589790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa02a36a-b384-4e3c-b8e4-582e82eba494 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:43.596811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 859/1737] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:43.597792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:43.605146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f8e1f8-844a-4174-a692-0d6eeee8d9bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:43.607173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f8e1f8-844a-4174-a692-0d6eeee8d9bd admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:43.607617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f8e1f8-844a-4174-a692-0d6eeee8d9bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:43.608944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f8e1f8-844a-4174-a692-0d6eeee8d9bd admin admin] PATH_INFO: `/v3/projects/2154c24a13c848c4b2522770855853f6/users/780e4a5c369a428c922436506ea3b895/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:43.627182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d5bde4-749d-4ae1-a4a2-314fdc1ff324 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:43.639745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:43.640407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:43.640706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:43.640987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:43.641687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:43.649571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:43.657004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f8e1f8-844a-4174-a692-0d6eeee8d9bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=2154c24a13c848c4b2522770855853f6, user_id=780e4a5c369a428c922436506ea3b895, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:43.665469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f8e1f8-844a-4174-a692-0d6eeee8d9bd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:43.704125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 879/1738] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:43 2022] PUT /identity/v3/projects/2154c24a13c848c4b2522770855853f6/users/780e4a5c369a428c922436506ea3b895/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:43.706570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:43.706898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:43.707513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:43.744771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 860/1739] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:43 2022] POST /identity/v3/projects => generated 381 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:43.745752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:43.830595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:43.832892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:43.883721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:43.968910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 880/1740] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:04:43 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:43.972914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.017933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ae3f0-dafc-4d04-9f82-10f0474dfca5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.033942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2582d5-e7ec-40f5-b393-5d1ed2f7d4be None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.049460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.050414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.052569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.053125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.053945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.054284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.055214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.056599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.056914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.057626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.064292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.064715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.095069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 861/1741] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:43 2022] GET /identity/v3/roles => generated 1420 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:44.105890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.128849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc9f4d0-67cd-4df6-82fe-be6ed45875d5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.129895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 881/1742] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:43 2022] POST /identity/v3/users => generated 291 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:44.133483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.138529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee43f876-e944-453e-8c33-871d3c28a8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.139688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee43f876-e944-453e-8c33-871d3c28a8a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.140187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee43f876-e944-453e-8c33-871d3c28a8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.140660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee43f876-e944-453e-8c33-871d3c28a8a1 admin admin] PATH_INFO: `/v3/projects/2154c24a13c848c4b2522770855853f6/users/780e4a5c369a428c922436506ea3b895/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.150401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee43f876-e944-453e-8c33-871d3c28a8a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2154c24a13c848c4b2522770855853f6, user_id=780e4a5c369a428c922436506ea3b895, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.153840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee43f876-e944-453e-8c33-871d3c28a8a1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.161843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645a36a2-343f-4726-a579-2afcd7dd3890 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.181145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 862/1743] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:44 2022] PUT /identity/v3/projects/2154c24a13c848c4b2522770855853f6/users/780e4a5c369a428c922436506ea3b895/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:44.192578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.192915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.193243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.193402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.195117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.195643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.197058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.199766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.203106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.239774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 882/1744] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:44 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:44.242544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] MFA Rules not processed for user `780e4a5c369a428c922436506ea3b895`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:44.242751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.255086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:44.267111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec2276-8327-4678-9524-1d67cd1db9ec admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.358825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:44.371740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.373368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 863/1745] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:44 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:04:44.374250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.377458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.377828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.378501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.379564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.382144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.385184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:44.407339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 883/1746] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:44 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:44.408451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.414312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:44.530218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754a7661-0072-433d-80bb-a8648e3513da admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.535782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.536807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.538567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.539148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.563486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.575727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.589090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c8d37e-38c0-414a-a3a9-3bd643ef5e27 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.610889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.611717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.612009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.626559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 884/1747] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:44 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:44.629700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.642755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.656546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62025ac3-6fe7-438a-a690-41443d046335 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:44.681233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1df8fa-7074-4dd7-9344-7f668f36c974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:44.682526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1df8fa-7074-4dd7-9344-7f668f36c974 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.683785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1df8fa-7074-4dd7-9344-7f668f36c974 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.684061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1df8fa-7074-4dd7-9344-7f668f36c974 admin admin] PATH_INFO: `/v3/projects/78d42b826c854caf90b655fa041914f1/users/fd36b0da8f0745e5b97eaa1476fad465/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.735300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1df8fa-7074-4dd7-9344-7f668f36c974 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78d42b826c854caf90b655fa041914f1, user_id=fd36b0da8f0745e5b97eaa1476fad465, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.738686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1df8fa-7074-4dd7-9344-7f668f36c974 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.771615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:44.779829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 885/1748] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:44 2022] PUT /identity/v3/projects/78d42b826c854caf90b655fa041914f1/users/fd36b0da8f0745e5b97eaa1476fad465/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:44.794727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:44.795023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:44.795308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:44.908449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] MFA Rules not processed for user `fd36b0da8f0745e5b97eaa1476fad465`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:44.913380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:44.944545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:44.946438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:44.989837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 864/1749] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:04:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 616 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:04:44.990780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:44.999690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:45.076942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f5c02-97ab-4dfc-be30-4a9a51d40737 service swift] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:45.185686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 886/1750] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:44 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:45.188162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:45.341944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:45.344165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.348344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.348957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] PATH_INFO: `/v3/projects/2ee71720233040ccbc5b7d058c832dd1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.355746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ee71720233040ccbc5b7d058c832dd1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:45.376097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:45.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:45.417866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc83fae-4817-458a-ba50-49e5c90dd93f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:45.435903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:45.436561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.436882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.437159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.444530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:45.449517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:45.528023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 887/1751] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:45 2022] POST /identity/v3/projects => generated 381 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:45.533088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:45.580249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ee71720233040ccbc5b7d058c832dd1', 'request_id': 'req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:45.600231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ee71720233040ccbc5b7d058c832dd1', 'request_id': 'req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:45.632690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ee71720233040ccbc5b7d058c832dd1', 'request_id': 'req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:45.671125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] The token cache is being invalidate because project 2ee71720233040ccbc5b7d058c832dd1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:45.674730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:45.675807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:45.683286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 865/1752] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:44 2022] DELETE /identity/v3/projects/2ee71720233040ccbc5b7d058c832dd1 => generated 0 bytes in 693 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:45.684553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:45.687394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:45.732656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674dfb9-d1d7-4627-8038-7e978ecf757f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:45.749252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583bb593-fed6-4c19-a461-77e71564e778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:45.749967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bb593-fed6-4c19-a461-77e71564e778 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.750303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bb593-fed6-4c19-a461-77e71564e778 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.750715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583bb593-fed6-4c19-a461-77e71564e778 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.751431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583bb593-fed6-4c19-a461-77e71564e778 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:45.753498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583bb593-fed6-4c19-a461-77e71564e778 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:45.792235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 888/1753] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:45 2022] POST /identity/v3/users => generated 297 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:45.793910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.794204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.794490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.815043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:45.832970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:45.834245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:45.856191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 889/1754] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:45.857120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:45.860570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7f606-ddd0-4b69-a9fe-da60a32d16e6 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:45.862868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:45.879624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:45.883478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.885509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.887033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] PATH_INFO: `/v3/users/9966cbd5be744463b35ee84f5dd655cf` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.888165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9e38c0-0c02-4f2c-991e-9a62644f3507 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:45.890572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] RBAC: Authorizing `identity:delete_user(user_id=9966cbd5be744463b35ee84f5dd655cf)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:45.896647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae038de6-194b-40ae-a703-8ed1fbf8ebbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:45.897376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae038de6-194b-40ae-a703-8ed1fbf8ebbe admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.897668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae038de6-194b-40ae-a703-8ed1fbf8ebbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.897949 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae038de6-194b-40ae-a703-8ed1fbf8ebbe admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.898716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae038de6-194b-40ae-a703-8ed1fbf8ebbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:45.900201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:45.900333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae038de6-194b-40ae-a703-8ed1fbf8ebbe admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:45.942375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 890/1755] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:45 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:45.951592 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:45.952013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:45.952294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:45.989806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:45.991678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:46.034338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 891/1756] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:46.036845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:46.061038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077c7214-83c8-419b-826a-3fb3778adb78 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:46.065204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9966cbd5be744463b35ee84f5dd655cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:46.077442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:46.079979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.080763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.081246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] PATH_INFO: `/v3/projects/b3ca13f3a22247c5a5ae89d9f16810b0/users/bbe0de9bcd3e4034ae5246a560059ac1/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.084773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9966cbd5be744463b35ee84f5dd655cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:46.105245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9966cbd5be744463b35ee84f5dd655cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:46.155880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3ca13f3a22247c5a5ae89d9f16810b0, user_id=bbe0de9bcd3e4034ae5246a560059ac1, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:46.169511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:46.185339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9966cbd5be744463b35ee84f5dd655cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:46.188751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9966cbd5be744463b35ee84f5dd655cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:46.210436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 892/1757] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:46 2022] PUT /identity/v3/projects/b3ca13f3a22247c5a5ae89d9f16810b0/users/bbe0de9bcd3e4034ae5246a560059ac1/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:46.228818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 866/1758] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:45 2022] DELETE /identity/v3/users/9966cbd5be744463b35ee84f5dd655cf => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:46.230046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:46.235989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:46.239293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:46.241811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:46.452796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd88bc-950e-4777-a955-5003ecce864a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:46.478988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea85ecf-0b64-4f4f-adb9-6f3839c4a5f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:46.479696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea85ecf-0b64-4f4f-adb9-6f3839c4a5f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.479986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea85ecf-0b64-4f4f-adb9-6f3839c4a5f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.480263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea85ecf-0b64-4f4f-adb9-6f3839c4a5f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.481045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea85ecf-0b64-4f4f-adb9-6f3839c4a5f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:46.482679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea85ecf-0b64-4f4f-adb9-6f3839c4a5f4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:46.486156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffa84aa-af03-4bc3-a89d-438195b8d6ac admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:46.493660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:46.494599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.495612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.496047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.498328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:46.499165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:46.501015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:46.542877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 867/1759] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:46 2022] GET /identity/v3/roles => generated 1420 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:46.545048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 893/1760] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:46.546216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.547672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:46.548332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.548790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.571277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b50e1b-c12e-4a3c-9f4d-2f043085608e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:46.587684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bcc9dc-7610-4b8c-903d-0e72bc829216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:46.591406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bcc9dc-7610-4b8c-903d-0e72bc829216 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.591671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bcc9dc-7610-4b8c-903d-0e72bc829216 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.592109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bcc9dc-7610-4b8c-903d-0e72bc829216 admin admin] PATH_INFO: `/v3/projects/b3ca13f3a22247c5a5ae89d9f16810b0/users/bbe0de9bcd3e4034ae5246a560059ac1/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.600856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bcc9dc-7610-4b8c-903d-0e72bc829216 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3ca13f3a22247c5a5ae89d9f16810b0, user_id=bbe0de9bcd3e4034ae5246a560059ac1, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:46.608662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:46.609980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:46.613285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bcc9dc-7610-4b8c-903d-0e72bc829216 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:46.637854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 868/1761] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:46.638768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:46.646274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 894/1762] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:46 2022] PUT /identity/v3/projects/b3ca13f3a22247c5a5ae89d9f16810b0/users/bbe0de9bcd3e4034ae5246a560059ac1/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:46.657619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.657951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.658357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.672054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66df02d4-6ab4-4a05-8205-58135fc70ba6 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:46.699571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:46.782298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] MFA Rules not processed for user `bbe0de9bcd3e4034ae5246a560059ac1`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:46.786544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:46.868257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:46.894116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:46.894814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:46.895105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:46.895383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:46.896094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:46.898604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:46.964870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 869/1763] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:46 2022] POST /identity/v3/projects => generated 363 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:46.967851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:47.055779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 895/1764] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:46 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 399 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:47.069252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:47.071425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:47.276336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d4aadd-5c41-47ea-bee5-98104f02fd73 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:47.307912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a8071-f36c-4563-8b79-3be1865bccb2 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:47.341112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:47.342014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:47.349097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:47.349530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:47.351206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:47.362617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:47.458851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:47.460067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:47.460508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:47.460803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:47.466469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 870/1765] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:46 2022] POST /identity/v3/users => generated 282 bytes in 499 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:04:47.467421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:47.486533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:47.674573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2591fe8d-383e-4de7-801e-5da9268e9057 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:47.686806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:47.689007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:47.690879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:47.691736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:47.702904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:47.717554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:47.718988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:47.720998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:47.723534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:47.741822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 871/1766] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:47.743325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:47.757457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a61e4b-d2c0-4bc0-9810-c98a75c7a824 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:47.778101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10775abf-97dd-4b4b-acc0-1a9b8174cac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:47.778971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10775abf-97dd-4b4b-acc0-1a9b8174cac5 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:47.779439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10775abf-97dd-4b4b-acc0-1a9b8174cac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:47.779897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10775abf-97dd-4b4b-acc0-1a9b8174cac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:47.780857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10775abf-97dd-4b4b-acc0-1a9b8174cac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:47.782642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10775abf-97dd-4b4b-acc0-1a9b8174cac5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:47.810330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 896/1767] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:47 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 742 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:04:47.815908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 872/1768] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:47 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:47.818517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:47.843694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41253123-8d29-4622-9f37-7d1f53256831 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:47.879237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be282bb0-710d-4923-a618-0740848be1be None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:47.879588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be282bb0-710d-4923-a618-0740848be1be None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:47.879875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be282bb0-710d-4923-a618-0740848be1be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:47.903579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:47.908826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:47.910747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:47.911037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:47.926997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:47.934026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:47.981347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-be282bb0-710d-4923-a618-0740848be1be None None] MFA Rules not processed for user `246dc188d9a44b1cb38721e3dd8a863a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:47.984125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-be282bb0-710d-4923-a618-0740848be1be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:48.004322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 897/1769] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:47 2022] GET /identity/v3/roles => generated 1420 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:48.006466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:48.012623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:48.057302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41af40f8-5b71-426f-8de9-95026a560823 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:48.073261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:48.074214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:48.074566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:48.074853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] PATH_INFO: `/v3/projects/de67cc0666ca41b68958a407a0f989c9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:48.079431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] RBAC: Authorizing `identity:delete_project(project_id=de67cc0666ca41b68958a407a0f989c9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:48.088062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:48.217468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 873/1770] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:04:48.218430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be282bb0-710d-4923-a618-0740848be1be None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:48.254612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be282bb0-710d-4923-a618-0740848be1be None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:48.273045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:48.276169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de67cc0666ca41b68958a407a0f989c9', 'request_id': 'req-c3597a2e-9476-40b6-bf41-56020281a89f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:48.312411 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de67cc0666ca41b68958a407a0f989c9', 'request_id': 'req-c3597a2e-9476-40b6-bf41-56020281a89f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:48.329942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de67cc0666ca41b68958a407a0f989c9', 'request_id': 'req-c3597a2e-9476-40b6-bf41-56020281a89f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:48.383437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] The token cache is being invalidate because project de67cc0666ca41b68958a407a0f989c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:48.383738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:48.384590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:48.387092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 898/1771] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:48 2022] DELETE /identity/v3/projects/de67cc0666ca41b68958a407a0f989c9 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:48.387721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:48.392358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:48.453919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f59198-3c78-4b59-b2b2-03ef323ab4fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:48.455521 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f59198-3c78-4b59-b2b2-03ef323ab4fc admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:48.456077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f59198-3c78-4b59-b2b2-03ef323ab4fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:48.456416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f59198-3c78-4b59-b2b2-03ef323ab4fc admin admin] PATH_INFO: `/v3/projects/43ac0fec8c6e4c50921ff8908093b725/users/8f188b7f8c7a4720b1aa3fdf8969c86d/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:48.486427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f59198-3c78-4b59-b2b2-03ef323ab4fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=43ac0fec8c6e4c50921ff8908093b725, user_id=8f188b7f8c7a4720b1aa3fdf8969c86d, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:48.490535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f59198-3c78-4b59-b2b2-03ef323ab4fc admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:48.528481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 874/1772] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:48 2022] PUT /identity/v3/projects/43ac0fec8c6e4c50921ff8908093b725/users/8f188b7f8c7a4720b1aa3fdf8969c86d/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:48.530038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382be7e2-bd42-4e7b-8270-8c25750929e4 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:48.530934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382be7e2-bd42-4e7b-8270-8c25750929e4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:48.531899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382be7e2-bd42-4e7b-8270-8c25750929e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:48.608801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-382be7e2-bd42-4e7b-8270-8c25750929e4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:48.617453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-382be7e2-bd42-4e7b-8270-8c25750929e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:48.624960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3597a2e-9476-40b6-bf41-56020281a89f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:48.856989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:48.857922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:48.858377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:48.858808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:48.860965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:48.870563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 875/1773] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:48.873818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:48.882523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:48.883316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:48.951749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] MFA Rules not processed for user `8f188b7f8c7a4720b1aa3fdf8969c86d`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:48.954008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:48.968330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:49.045028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.046548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723450cf-1596-4ceb-b10f-e27b765727a3 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.081812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 899/1774] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:48 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 694 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:49.083368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.084982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.085387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.126006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:49.136240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:49.148760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 900/1775] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:49.152746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.171892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 876/1776] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:48 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:49.172134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198610f-06de-4b58-bc63-cbf2e25a7d7c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.173838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.179160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:49.193214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.194165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.194639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.195165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.196321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:49.198025 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.201797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.235070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66897286-1103-4cf4-a046-42047b4673b4 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.242131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 901/1777] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:49.246311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.251412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.252105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.252392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.252672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.257410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.265791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bd8e8a-7589-4e95-9d2b-62cb9f0a320f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.268274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.279915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.282765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.283083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.284743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.285578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.293069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.338524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:49.365119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 877/1778] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:49 2022] POST /identity/v3/projects => generated 365 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:49.367798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.440887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 902/1779] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:49 2022] POST /identity/v3/projects => generated 389 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:04:49.443386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.651432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49f5408-068f-4011-9f24-572d5aa02544 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.658008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.658854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.659493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.659811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.660576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.664108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.713502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 903/1780] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:49 2022] POST /identity/v3/users => generated 295 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:49.718982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.719563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947be695-6689-4e7f-9207-25a0c859f536 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.747034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.749031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.750151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22191df7-9540-4ea1-b5dc-396ffd950cb5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.750834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.751144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.753284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.757416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.762937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.763661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.763955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.764235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.770245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.773782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.787067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 878/1781] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:49 2022] POST /identity/v3/users => generated 289 bytes in 420 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:49.792723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.823912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938cb26-39fd-478f-bcdd-6d80573d3269 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.824723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 904/1782] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:49 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:49.830483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.839803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.840546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.840890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.841188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.842186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.844057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.855928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02820e28-fb82-4913-be48-72fb5df65e83 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.869453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 879/1783] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:49 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:49.870176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.870913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.871198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.871491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.873018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.874158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.875940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:49.904810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc3d625-5595-4b9e-9766-7d7f032bbf20 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.911329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 905/1784] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:49 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:49.915299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:49.923920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.925144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.925595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.925991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] PATH_INFO: `/v3/projects/14795235b6a542e49556b922d872aa71/users/a58460584e37489083803f5c026abb58/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.944771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9236776-61a8-4c23-8dcb-e15b6ec3678a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:49.956474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566d2275-d08b-4b98-9a28-c0ccf9d73fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:49.959526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566d2275-d08b-4b98-9a28-c0ccf9d73fb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:49.961895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566d2275-d08b-4b98-9a28-c0ccf9d73fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:49.962634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566d2275-d08b-4b98-9a28-c0ccf9d73fb0 admin admin] PATH_INFO: `/v3/projects/749fc4329eef462e94858ab572ef9d50/users/a5476642950343f6b4f598b91d1dd10f/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:49.969641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14795235b6a542e49556b922d872aa71, user_id=a58460584e37489083803f5c026abb58, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:49.973557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:50.003926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 880/1785] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:49 2022] PUT /identity/v3/projects/14795235b6a542e49556b922d872aa71/users/a58460584e37489083803f5c026abb58/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:50.004953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566d2275-d08b-4b98-9a28-c0ccf9d73fb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=749fc4329eef462e94858ab572ef9d50, user_id=a5476642950343f6b4f598b91d1dd10f, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:50.009021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:50.010912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566d2275-d08b-4b98-9a28-c0ccf9d73fb0 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:50.071467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 906/1786] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:49 2022] PUT /identity/v3/projects/749fc4329eef462e94858ab572ef9d50/users/a5476642950343f6b4f598b91d1dd10f/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:50.082566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:50.083340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:50.084007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:50.185075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] MFA Rules not processed for user `a5476642950343f6b4f598b91d1dd10f`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:50.187049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:50.242982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96b9d9-3cfa-44ee-8347-34dd52e872e8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:50.275552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:50.376056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 907/1787] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:50 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:04:50.378246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:50.412618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:50.413488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:50.413847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:50.414186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:50.414985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:50.419209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:50.426746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:50.456762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 881/1788] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:50 2022] GET /identity/v3/roles => generated 1420 bytes in 445 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:50.459014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:50.479572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b8030e-81f1-49a4-9577-17b95dd44f8c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:50.499057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca8e6cf-6ac6-423b-ab2f-c5325adab111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:50.502092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca8e6cf-6ac6-423b-ab2f-c5325adab111 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:50.502396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca8e6cf-6ac6-423b-ab2f-c5325adab111 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:50.502710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca8e6cf-6ac6-423b-ab2f-c5325adab111 admin admin] PATH_INFO: `/v3/projects/14795235b6a542e49556b922d872aa71/users/a58460584e37489083803f5c026abb58/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:50.514478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca8e6cf-6ac6-423b-ab2f-c5325adab111 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14795235b6a542e49556b922d872aa71, user_id=a58460584e37489083803f5c026abb58, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:50.524547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca8e6cf-6ac6-423b-ab2f-c5325adab111 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:50.554204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf395ace-d4b5-45ba-899e-25bcedd88875 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:50.579904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 882/1789] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:50 2022] PUT /identity/v3/projects/14795235b6a542e49556b922d872aa71/users/a58460584e37489083803f5c026abb58/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:50.592025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:50.592763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:50.596168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:50.695893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] MFA Rules not processed for user `a58460584e37489083803f5c026abb58`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:50.698498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:50.744237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b092f0a0-2d8e-4270-9cbf-3d5454b07bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:50.747474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b092f0a0-2d8e-4270-9cbf-3d5454b07bf9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:50.749735 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b092f0a0-2d8e-4270-9cbf-3d5454b07bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:50.750036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b092f0a0-2d8e-4270-9cbf-3d5454b07bf9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:50.751755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b092f0a0-2d8e-4270-9cbf-3d5454b07bf9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:50.759402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b092f0a0-2d8e-4270-9cbf-3d5454b07bf9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:50.847600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 908/1790] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:50 2022] POST /identity/v3/projects => generated 387 bytes in 467 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:50.848682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:50.849061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:50.849351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:50.914357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:50.915614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:50.939002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 883/1791] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:50 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:50.940177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:50.960659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db18a7a0-8da8-4988-b801-e2daeab4e6b4 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.046547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 909/1792] 10.222.0.27 () {58 vars in 1015 bytes} [Tue Jul 19 11:04:50 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:51.047486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:51.075283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.076393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.076393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.076604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.077323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.079302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.079886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:51.128680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 884/1793] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:50 2022] POST /identity/v3/users => generated 300 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:51.128680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:51.135522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:51.176899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be474d0-f928-4b0a-a772-6bba5e1a5e17 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.198015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.198977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.199286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.199565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.233901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.235577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.267796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 885/1794] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:51 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:51.268757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:51.286456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88aab1e-c301-4df0-b3bf-755eb4de3bf0 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.292901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.293631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.293919 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.294195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.294938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.296803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.311557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 886/1795] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:51.316509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:51.322006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70515ab-4c75-48f2-93e4-28997e79f8c6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.339354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264ea20-7e28-4e86-8dee-5e4d7428ad7b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.346926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.349242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.349555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.349838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] PATH_INFO: `/v3/projects/0e2f57d1d7ae41309fe8958364f5bcb7/users/1987d7135c264b439efdcb96e61091cc/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.368255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.376256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.376583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.376852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.426264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e2f57d1d7ae41309fe8958364f5bcb7, user_id=1987d7135c264b439efdcb96e61091cc, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.454811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.499389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 887/1796] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:51 2022] PUT /identity/v3/projects/0e2f57d1d7ae41309fe8958364f5bcb7/users/1987d7135c264b439efdcb96e61091cc/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:51.504775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:51.555053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:51.716536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.718444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.757645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 910/1797] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:51 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 710 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:51.797627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeb3a10-544d-46af-9927-fb6aa7ffafec admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.810792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a73c47-16af-44ae-a93e-a0ba50f47089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.812270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a73c47-16af-44ae-a93e-a0ba50f47089 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.813075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a73c47-16af-44ae-a93e-a0ba50f47089 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.813351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a73c47-16af-44ae-a93e-a0ba50f47089 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.815408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a73c47-16af-44ae-a93e-a0ba50f47089 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.823467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a73c47-16af-44ae-a93e-a0ba50f47089 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.849510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 888/1798] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:51.855148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:51.881391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8d1c23-f267-489f-85fb-273ebf80592d service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:51.894760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:51.895678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.895965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.896240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] PATH_INFO: `/v3/projects/0e2f57d1d7ae41309fe8958364f5bcb7/users/1987d7135c264b439efdcb96e61091cc/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:51.909661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e2f57d1d7ae41309fe8958364f5bcb7, user_id=1987d7135c264b439efdcb96e61091cc, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:51.938312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:51.981885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 911/1799] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:51 2022] PUT /identity/v3/projects/0e2f57d1d7ae41309fe8958364f5bcb7/users/1987d7135c264b439efdcb96e61091cc/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:51.993951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:51.994479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:51.994814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:52.026153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:52.105002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] MFA Rules not processed for user `1987d7135c264b439efdcb96e61091cc`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:52.107450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:52.258122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506d50bb-065e-4d6a-b454-ce6e14c8178e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:52.268920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:52.277879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:52.278145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:52.278431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:52.402105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 889/1800] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:51 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 416 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:52.402105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:52.410332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:52.508797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:52.518286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:52.542017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 912/1801] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:52 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 516 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:04:52.544339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:52.546423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:52.555862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:52.568260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d21839-ea64-4058-9517-053e5f1e23d5 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:52.576840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:52.584420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:52.585229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:52.585718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:52.600214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:52.603875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:52.627077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 913/1802] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:52 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:52.703058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c08f978-2b7a-4261-8f33-f1b2b79c55a0 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:52.723313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:52.726387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:52.726751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:52.727034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] PATH_INFO: `/v3/projects/a591c3a3a0694cf1a18b4a8b8e1bb2e8` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:52.730541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=a591c3a3a0694cf1a18b4a8b8e1bb2e8)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:52.733807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:52.920533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:52.993477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a591c3a3a0694cf1a18b4a8b8e1bb2e8', 'request_id': 'req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.029992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a591c3a3a0694cf1a18b4a8b8e1bb2e8', 'request_id': 'req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.047229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a591c3a3a0694cf1a18b4a8b8e1bb2e8', 'request_id': 'req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.070390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] The token cache is being invalidate because project a591c3a3a0694cf1a18b4a8b8e1bb2e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:53.071088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.071946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.078919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:53.079620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 890/1803] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:52 2022] DELETE /identity/v3/projects/a591c3a3a0694cf1a18b4a8b8e1bb2e8 => generated 0 bytes in 673 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:53.093230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d256e9-5493-4321-968e-861c2a3e38bd service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:53.106753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:53.108288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:53.210068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:53.212220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:53.212584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:53.213247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] PATH_INFO: `/v3/users/780e4a5c369a428c922436506ea3b895` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:53.217479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] RBAC: Authorizing `identity:delete_user(user_id=780e4a5c369a428c922436506ea3b895)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:53.222007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:53.305485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e242c31-bd45-4ee3-bb82-4e90b1ad41ea admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:53.333445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:53.334876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:53.335165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:53.335439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] PATH_INFO: `/v3/users/a5476642950343f6b4f598b91d1dd10f` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:53.341946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5476642950343f6b4f598b91d1dd10f)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:53.345161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:53.387295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '780e4a5c369a428c922436506ea3b895'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.404357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '780e4a5c369a428c922436506ea3b895'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.454711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '780e4a5c369a428c922436506ea3b895'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.456439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '780e4a5c369a428c922436506ea3b895'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.458171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6e15b0e8-5b23-43b1-9980-4e058e18810b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '780e4a5c369a428c922436506ea3b895'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.522974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5476642950343f6b4f598b91d1dd10f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.527856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 914/1804] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:53 2022] DELETE /identity/v3/users/780e4a5c369a428c922436506ea3b895 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:53.550700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5476642950343f6b4f598b91d1dd10f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.564671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5476642950343f6b4f598b91d1dd10f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.590685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5476642950343f6b4f598b91d1dd10f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.601294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5476642950343f6b4f598b91d1dd10f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:53.611133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 891/1805] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:53 2022] DELETE /identity/v3/users/a5476642950343f6b4f598b91d1dd10f => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:53.616029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b347c7-afe8-4fbb-b7bd-2047102fa2d8 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:53.616323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b347c7-afe8-4fbb-b7bd-2047102fa2d8 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:53.616600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b347c7-afe8-4fbb-b7bd-2047102fa2d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:53.668147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a1b347c7-afe8-4fbb-b7bd-2047102fa2d8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:53.673036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a1b347c7-afe8-4fbb-b7bd-2047102fa2d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:53.930115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 915/1806] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:54.085947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:54.087558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:54.328345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:54.357995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c9a83-c4ff-4b28-bdaf-05f6458b911e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:54.371178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931b76c-d9fa-48fd-846e-f349628c60bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:54.372399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931b76c-d9fa-48fd-846e-f349628c60bb service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:54.372866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931b76c-d9fa-48fd-846e-f349628c60bb service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:54.373253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931b76c-d9fa-48fd-846e-f349628c60bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:54.394991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931b76c-d9fa-48fd-846e-f349628c60bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:54.396831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931b76c-d9fa-48fd-846e-f349628c60bb service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:54.426644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 892/1807] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:55.545616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b727c2-a8e7-444f-8ed1-7c223f2ca8a6 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:55.557473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b727c2-a8e7-444f-8ed1-7c223f2ca8a6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:55.557813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b727c2-a8e7-444f-8ed1-7c223f2ca8a6 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:55.561924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 916/1808] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:04:55 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:04:55.568964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:55.569502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:55.569750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:55.616365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:55.617710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:55.808862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 893/1809] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:04:55 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:55.824403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b727c2-a8e7-444f-8ed1-7c223f2ca8a6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:55.827027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-23b727c2-a8e7-444f-8ed1-7c223f2ca8a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:55.868632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b727c2-a8e7-444f-8ed1-7c223f2ca8a6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:55.875214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0fde40-85eb-4e69-9d8c-8147da2f37e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:55.876166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0fde40-85eb-4e69-9d8c-8147da2f37e8 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:55.876612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0fde40-85eb-4e69-9d8c-8147da2f37e8 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:55.877121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0fde40-85eb-4e69-9d8c-8147da2f37e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:55.889404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:55.898116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0fde40-85eb-4e69-9d8c-8147da2f37e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:55.903770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0fde40-85eb-4e69-9d8c-8147da2f37e8 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:55.932069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 917/1810] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:55 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:56.227455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:56.229512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.245484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b27695-4cea-43ce-aa68-17b4c1ed197c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:56.251473 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:56.252365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.252649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.252994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] PATH_INFO: `/v3/projects/2154c24a13c848c4b2522770855853f6` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.254579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=2154c24a13c848c4b2522770855853f6)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:56.257179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:56.328133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2154c24a13c848c4b2522770855853f6', 'request_id': 'req-efb85913-add3-486f-8f4e-26b822349f9a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:56.336775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2154c24a13c848c4b2522770855853f6', 'request_id': 'req-efb85913-add3-486f-8f4e-26b822349f9a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:56.362058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2154c24a13c848c4b2522770855853f6', 'request_id': 'req-efb85913-add3-486f-8f4e-26b822349f9a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:56.385083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] The token cache is being invalidate because project 2154c24a13c848c4b2522770855853f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:56.385534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:56.386335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-efb85913-add3-486f-8f4e-26b822349f9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:56.388401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 894/1811] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:56 2022] DELETE /identity/v3/projects/2154c24a13c848c4b2522770855853f6 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:56.394138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a10d3f2-fb71-4f5f-a0e1-cb55bddcc1e2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.394566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a10d3f2-fb71-4f5f-a0e1-cb55bddcc1e2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.394959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a10d3f2-fb71-4f5f-a0e1-cb55bddcc1e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.422542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8459b78f-fd83-4fca-be9d-182b90eff3ab None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.423467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8459b78f-fd83-4fca-be9d-182b90eff3ab None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.423969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8459b78f-fd83-4fca-be9d-182b90eff3ab None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.425954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 895/1812] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:04:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 19 11:04:56.437827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-2a10d3f2-fb71-4f5f-a0e1-cb55bddcc1e2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:56.440252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2a10d3f2-fb71-4f5f-a0e1-cb55bddcc1e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.448421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e7a873-8638-44aa-85ef-07f4e8c8015f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.448608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e7a873-8638-44aa-85ef-07f4e8c8015f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.448938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e7a873-8638-44aa-85ef-07f4e8c8015f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.484961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-61e7a873-8638-44aa-85ef-07f4e8c8015f None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:56.486304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-61e7a873-8638-44aa-85ef-07f4e8c8015f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.581324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 918/1813] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:56.586304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.586304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.586304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.620377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] MFA Rules not processed for user `b8f2a78f3a3e4b8e8937e3bc39524690`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:56.622268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.639457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:56.665078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 896/1814] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:04:56 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:56.666489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.666977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.667364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.690639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:56.692361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.730761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 897/1815] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:56.731937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:56.733772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.739683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 919/1816] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:56.740965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:56.742844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:56.775555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b03a773-558f-4467-8b1d-22a48285db2c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:56.782300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:56.783582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.784080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.784591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] PATH_INFO: `/v3/projects/749fc4329eef462e94858ab572ef9d50` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.786383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] RBAC: Authorizing `identity:delete_project(project_id=749fc4329eef462e94858ab572ef9d50)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:56.788852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:56.812853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06c8029-584a-4150-abe4-b059d079c90a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:56.842732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:56.843408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:56.843696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:56.844007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:56.894377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:56.897373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:56.920451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 920/1817] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:56.921892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:56.940111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af489cd4-4068-4305-9c41-f53e8e34516f service nova] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:56.953048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '749fc4329eef462e94858ab572ef9d50', 'request_id': 'req-737bce12-b2f5-469b-a78e-101b70f1e316'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:56.984125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '749fc4329eef462e94858ab572ef9d50', 'request_id': 'req-737bce12-b2f5-469b-a78e-101b70f1e316'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.012345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '749fc4329eef462e94858ab572ef9d50', 'request_id': 'req-737bce12-b2f5-469b-a78e-101b70f1e316'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.048575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:57.050156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:57.050639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:57.051511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:57.053286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:04:57.054008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:57.057386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:57.071340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] The token cache is being invalidate because project 749fc4329eef462e94858ab572ef9d50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:04:57.072818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.074499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.091194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 898/1818] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:04:56 2022] DELETE /identity/v3/projects/749fc4329eef462e94858ab572ef9d50 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:57.092341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:57.094168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:57.126259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 921/1819] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:04:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:04:57.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737bce12-b2f5-469b-a78e-101b70f1e316 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:57.136012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:57.141703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:57.146131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:57.147697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:57.148052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:57.148395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] PATH_INFO: `/v3/users/1987d7135c264b439efdcb96e61091cc` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:57.151215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1987d7135c264b439efdcb96e61091cc)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:57.156914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:57.186805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6e0be7-bb82-4ab0-a93f-2bc087ce3984 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:57.204203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:57.205405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:57.208672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:57.209842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:57.210819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:57.223946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:57.285114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1987d7135c264b439efdcb96e61091cc'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.290157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:57.301681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1987d7135c264b439efdcb96e61091cc'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.308922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 922/1820] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:57 2022] POST /identity/v3/projects => generated 383 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:57.312613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:57.320544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1987d7135c264b439efdcb96e61091cc'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.358775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1987d7135c264b439efdcb96e61091cc'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.360383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1987d7135c264b439efdcb96e61091cc'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:04:57.380732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 899/1821] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:04:57 2022] DELETE /identity/v3/users/1987d7135c264b439efdcb96e61091cc => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:57.579113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af5b704-9d5b-4c4c-aa6c-dcd0dbfe761f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:57.727525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:57.731177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:57.731882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:57.734002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:57.736115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:57.747636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:57.825773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:57.826538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 923/1822] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:57 2022] POST /identity/v3/users => generated 292 bytes in 513 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:57.827713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:57.852274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf51eb0c-4c32-41e3-9981-96528dd6b014 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:57.868961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:57.869654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:57.869942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:57.870223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:57.871021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:57.872879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:57.907682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 900/1823] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:57 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:57.909075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:57.938604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a682383a-823a-4664-a4b0-5ddda5b72c72 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:57.954285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dbabe8-0037-4aa3-a55a-3147b3eaf610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:57.955125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dbabe8-0037-4aa3-a55a-3147b3eaf610 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:57.955518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dbabe8-0037-4aa3-a55a-3147b3eaf610 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:57.955901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dbabe8-0037-4aa3-a55a-3147b3eaf610 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:57.956908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5dbabe8-0037-4aa3-a55a-3147b3eaf610 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:57.958811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5dbabe8-0037-4aa3-a55a-3147b3eaf610 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:57.978899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:57.979424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 924/1824] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:57 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:57.990282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814414e8-7b97-4351-af70-93f2f1dc00b5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:58.001381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:58.004466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:58.005240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:58.005240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] PATH_INFO: `/v3/projects/41dedac5da8049af8b05a0e95f530cef/users/d555138078f64cb6a673db7dddc9739d/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:58.048002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] RBAC: Authorizing `identity:create_grant(project_id=41dedac5da8049af8b05a0e95f530cef, user_id=d555138078f64cb6a673db7dddc9739d, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:58.054363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:58.080457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 901/1825] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:57 2022] PUT /identity/v3/projects/41dedac5da8049af8b05a0e95f530cef/users/d555138078f64cb6a673db7dddc9739d/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:58.093443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:58.093860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:58.094257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:58.161709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] MFA Rules not processed for user `d555138078f64cb6a673db7dddc9739d`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:58.163875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:58.367855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:58.375412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:58.376229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 925/1826] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:58.519736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d92142-23a5-488b-845d-d8014848812d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:58.532006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:58.536158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:58.536620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:58.536935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:58.538333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:58.540300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:58.601227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:58.608280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 902/1827] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:04:58 2022] POST /identity/v3/projects => generated 383 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:58.616012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac4e7bd-b2e5-4293-af8f-1ad809b8f6c3 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:58.650241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:58.818107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:58.822817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:58.823333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:58.823868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:58.825072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:58.827810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:58.869682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 926/1828] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:04:58 2022] POST /identity/v3/users => generated 298 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:04:58.874313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:58.897562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd2f9e-f0c9-4060-94c9-d773b6760149 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:58.920523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:58.922663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:58.923214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:58.924135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:58.925550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:58.929554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:58.977776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 903/1829] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:58.982765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:58.999658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2857852-322c-4aab-8c6a-a29f40a313fe admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:59.011639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:59.013918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:59.014322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:59.014681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] PATH_INFO: `/v3/projects/3a3b39a635a5448d97002fcd4af48418/users/4979d400ef55493d8fb9acc279fb882d/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:59.059730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a3b39a635a5448d97002fcd4af48418, user_id=4979d400ef55493d8fb9acc279fb882d, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:59.062972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:59.087673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 927/1830] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:58 2022] PUT /identity/v3/projects/3a3b39a635a5448d97002fcd4af48418/users/4979d400ef55493d8fb9acc279fb882d/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:59.092611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:59.106916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cd8194-1340-4a4d-895b-2cdad6d1f501 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:59.328911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e06593-b372-474c-9589-2078b581902d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:59.329640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e06593-b372-474c-9589-2078b581902d admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:59.329924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e06593-b372-474c-9589-2078b581902d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:59.330216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e06593-b372-474c-9589-2078b581902d admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:59.330935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e06593-b372-474c-9589-2078b581902d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:59.338261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e06593-b372-474c-9589-2078b581902d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:59.368950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 904/1831] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:04:59 2022] GET /identity/v3/roles => generated 1420 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:04:59.370173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:59.388649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e1ab9d-0cde-4741-9399-6b98e5f4657f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:59.402962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:59.403910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:59.404198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:59.404477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] PATH_INFO: `/v3/projects/3a3b39a635a5448d97002fcd4af48418/users/4979d400ef55493d8fb9acc279fb882d/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:59.416998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a3b39a635a5448d97002fcd4af48418, user_id=4979d400ef55493d8fb9acc279fb882d, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:59.421077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:59.450764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 928/1832] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:04:59 2022] PUT /identity/v3/projects/3a3b39a635a5448d97002fcd4af48418/users/4979d400ef55493d8fb9acc279fb882d/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:04:59.461299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:59.461627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:59.461838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:59.543598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] MFA Rules not processed for user `4979d400ef55493d8fb9acc279fb882d`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:04:59.548946 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:59.684225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:04:59.693022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 905/1833] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:04:59 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:04:59.713000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:04:59.716439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:04:59.790380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83227500-47cf-4157-9995-4fa577cb6f35 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:04:59.798978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:04:59.799718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:04:59.800164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:04:59.800568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:04:59.826110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:04:59.829201 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:04:59.852682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 929/1834] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:59 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:04:59.993027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:00.004397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4514b3a7-31ff-4f76-b778-5bb580eb4364 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:00.009976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:00.011049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:00.011574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:00.012286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:00.124826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:00.126652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:00.153222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 906/1835] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:04:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:00.833103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:00.906685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:00.908252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:01.032891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6e4fd8-ddb2-4486-908b-e6945b528fea service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.051619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.054329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.055108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.056185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] PATH_INFO: `/v3/users/d555138078f64cb6a673db7dddc9739d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.060179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d555138078f64cb6a673db7dddc9739d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.066383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.075046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ff24f-0422-44fe-abdf-03e93eb46f14 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.095834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.097400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.097701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.097980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] PATH_INFO: `/v3/projects/0e2f57d1d7ae41309fe8958364f5bcb7` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.102037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e2f57d1d7ae41309fe8958364f5bcb7)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.106950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.111231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:01.193141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd555138078f64cb6a673db7dddc9739d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.215089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd555138078f64cb6a673db7dddc9739d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.243435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd555138078f64cb6a673db7dddc9739d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.244203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd555138078f64cb6a673db7dddc9739d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.244856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-720972c6-24b4-4bb3-809f-abb4e1d28142 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd555138078f64cb6a673db7dddc9739d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.258789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e2f57d1d7ae41309fe8958364f5bcb7', 'request_id': 'req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.280323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 930/1836] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:00 2022] DELETE /identity/v3/users/d555138078f64cb6a673db7dddc9739d => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:01.284961 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e2f57d1d7ae41309fe8958364f5bcb7', 'request_id': 'req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.285494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.285690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.286050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.309947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:01.311172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:01.319718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e2f57d1d7ae41309fe8958364f5bcb7', 'request_id': 'req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.339942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] The token cache is being invalidate because project 0e2f57d1d7ae41309fe8958364f5bcb7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:01.340235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.340939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-28c3f418-09d7-4ec6-9e54-8a8fe60ee9f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:01.346752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 907/1837] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:00 2022] DELETE /identity/v3/projects/0e2f57d1d7ae41309fe8958364f5bcb7 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:01.348616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc70de-4ef9-4cb8-b253-f68ea7758c9a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.348956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc70de-4ef9-4cb8-b253-f68ea7758c9a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.349238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc70de-4ef9-4cb8-b253-f68ea7758c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.382799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-72bc70de-4ef9-4cb8-b253-f68ea7758c9a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:01.384643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-72bc70de-4ef9-4cb8-b253-f68ea7758c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:01.410012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 931/1838] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:01.417414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:01.429140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.431014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:01.485249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 908/1839] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:01.487166 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.487466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.487750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.510123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:01.511361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:01.514286 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ad63d4-a588-4ba6-aaf3-ef29027f7ef3 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.518150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.518796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.519077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.519422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.525789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 909/1840] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:01.530105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.531036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.532465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.540237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006941f-4eac-4eb7-b95f-36678fb77eb9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.542938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 932/1841] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:01.546291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.548733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.549037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.549375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.550061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:01.550628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.552002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.566640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 910/1842] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:05:01.571594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.583329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276ef4e0-aefc-490b-8187-0b4d230da3ae service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.588732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.589478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.589830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.590176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.590935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.592921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.626611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 933/1843] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:01 2022] POST /identity/v3/projects => generated 373 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:01.627767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.639652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539476e5-35e9-4cee-bfd2-3210b25438f0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.713418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.714612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.715409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.715813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.716937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.719927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.748745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 911/1844] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:01 2022] POST /identity/v3/users => generated 287 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:01.751318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.761737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbf1391-97ac-4470-bc72-f1e0b0b638de admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.766380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.767039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.767344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.767605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.768330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.769955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.775983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:01.782196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 934/1845] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:01.786182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.797243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaee8b4-4303-484e-a6b3-20e8376680e1 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.801985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.802686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.803011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.803290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.804029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.805660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.820925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 912/1846] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:01.822190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.840830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133cc7ae-0b57-4219-853f-035f91636e8c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.846661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8868d0-9223-42d8-87cc-75c65f0caee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.847672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8868d0-9223-42d8-87cc-75c65f0caee6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.848025 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8868d0-9223-42d8-87cc-75c65f0caee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.848309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8868d0-9223-42d8-87cc-75c65f0caee6 admin admin] PATH_INFO: `/v3/projects/a36afd3997cb4202aa69a2de07790383/users/7455597c67a2478eaf10b120c1629231/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.862030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:01.863845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:01.885180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8868d0-9223-42d8-87cc-75c65f0caee6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a36afd3997cb4202aa69a2de07790383, user_id=7455597c67a2478eaf10b120c1629231, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.887978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a7237-68a1-4b5e-9049-0a4ee45eb5c2 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:01.888110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8868d0-9223-42d8-87cc-75c65f0caee6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.898254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:01.899258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.899553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.899834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] PATH_INFO: `/v3/projects/41dedac5da8049af8b05a0e95f530cef` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.901395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=41dedac5da8049af8b05a0e95f530cef)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:01.903685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:01.947178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 935/1847] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:01 2022] PUT /identity/v3/projects/a36afd3997cb4202aa69a2de07790383/users/7455597c67a2478eaf10b120c1629231/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:01.958358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:01.958678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:01.958922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:01.998750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41dedac5da8049af8b05a0e95f530cef', 'request_id': 'req-310fc06c-acad-4b78-8493-b15d0ed5be7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.005604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41dedac5da8049af8b05a0e95f530cef', 'request_id': 'req-310fc06c-acad-4b78-8493-b15d0ed5be7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.010268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] MFA Rules not processed for user `7455597c67a2478eaf10b120c1629231`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:02.012055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.015158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41dedac5da8049af8b05a0e95f530cef', 'request_id': 'req-310fc06c-acad-4b78-8493-b15d0ed5be7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.036735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] The token cache is being invalidate because project 41dedac5da8049af8b05a0e95f530cef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:02.036735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.040824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.054685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 913/1848] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:01 2022] DELETE /identity/v3/projects/41dedac5da8049af8b05a0e95f530cef => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:02.059651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:02.061322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.077271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:02.109289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 936/1849] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:01 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:02.119742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:02.121437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.143903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310fc06c-acad-4b78-8493-b15d0ed5be7b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:02.149318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:02.150228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.150516 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.150796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] PATH_INFO: `/v3/users/4979d400ef55493d8fb9acc279fb882d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.152416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4979d400ef55493d8fb9acc279fb882d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:02.154728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:02.201871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2715154-c048-4630-8b96-4f1c2ff10eae None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:02.207803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:02.208531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.209731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.219258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:02.220832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:02.221960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4979d400ef55493d8fb9acc279fb882d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.231446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4979d400ef55493d8fb9acc279fb882d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.231694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 937/1850] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:02 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:02.241978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4979d400ef55493d8fb9acc279fb882d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.254235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:02.257548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.258020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4979d400ef55493d8fb9acc279fb882d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.258855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-95427932-b3ca-4476-b3a1-165d978bcc74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4979d400ef55493d8fb9acc279fb882d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.268714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 914/1851] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:02 2022] DELETE /identity/v3/users/4979d400ef55493d8fb9acc279fb882d => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:02.277036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362735e6-e6a7-4c72-ac48-e75b27bab4c1 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:02.338500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:02.339418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.339720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.340011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] PATH_INFO: `/v3/users/7455597c67a2478eaf10b120c1629231` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.341356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] RBAC: Authorizing `identity:delete_user(user_id=7455597c67a2478eaf10b120c1629231)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:02.343395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:02.355063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:02.398972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7455597c67a2478eaf10b120c1629231'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.406721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7455597c67a2478eaf10b120c1629231'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7455597c67a2478eaf10b120c1629231'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.415466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7455597c67a2478eaf10b120c1629231'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.416098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7455597c67a2478eaf10b120c1629231'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.436629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 938/1852] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:02 2022] DELETE /identity/v3/users/7455597c67a2478eaf10b120c1629231 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:02.441640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.442009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.442299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.461580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:02.462973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.558954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 915/1853] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:02.578058 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:02.579913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.610030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:02.611786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.639779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f68db8-2cf9-4c78-90af-1cf03dea48e5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:02.668875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:02.672093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.672480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.673605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] PATH_INFO: `/v3/projects/3a3b39a635a5448d97002fcd4af48418` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.676742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9f2ff-f750-4f7b-9570-ca8e312a9aca admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:02.677464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a3b39a635a5448d97002fcd4af48418)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:02.681077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:02.682212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20966de1-f20a-4fce-8730-8acc7095f46e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:02.685724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20966de1-f20a-4fce-8730-8acc7095f46e service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.685724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20966de1-f20a-4fce-8730-8acc7095f46e service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.685724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20966de1-f20a-4fce-8730-8acc7095f46e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.696166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20966de1-f20a-4fce-8730-8acc7095f46e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:02.709810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20966de1-f20a-4fce-8730-8acc7095f46e service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:02.736141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 939/1854] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:02.747038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:02.815129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a3b39a635a5448d97002fcd4af48418', 'request_id': 'req-60b6e560-6d42-499a-ac43-a1b366fd8bb3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.833560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a3b39a635a5448d97002fcd4af48418', 'request_id': 'req-60b6e560-6d42-499a-ac43-a1b366fd8bb3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.845193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a3b39a635a5448d97002fcd4af48418', 'request_id': 'req-60b6e560-6d42-499a-ac43-a1b366fd8bb3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.870501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] The token cache is being invalidate because project 3a3b39a635a5448d97002fcd4af48418 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:02.870939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.873077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:02.878309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 916/1855] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:02 2022] DELETE /identity/v3/projects/3a3b39a635a5448d97002fcd4af48418 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:02.880212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831dcc47-0788-43e0-8de0-0774057d2b7c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:02.881083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831dcc47-0788-43e0-8de0-0774057d2b7c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:02.881656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831dcc47-0788-43e0-8de0-0774057d2b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:02.905513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-831dcc47-0788-43e0-8de0-0774057d2b7c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:02.907794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-831dcc47-0788-43e0-8de0-0774057d2b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:02.926349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:02.927997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:03.020432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b6e560-6d42-499a-ac43-a1b366fd8bb3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.027666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.028924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.029353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.029754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] PATH_INFO: `/v3/users/fd36b0da8f0745e5b97eaa1476fad465` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.031165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd36b0da8f0745e5b97eaa1476fad465)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.033594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.123115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd36b0da8f0745e5b97eaa1476fad465'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:03.143359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd36b0da8f0745e5b97eaa1476fad465'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:03.159557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 940/1856] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:03.166603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.167233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.167568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.169189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd36b0da8f0745e5b97eaa1476fad465'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:03.203657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd36b0da8f0745e5b97eaa1476fad465'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:03.208176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c2f9e4d1-60a7-41ba-ac77-ebf92e90c5dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd36b0da8f0745e5b97eaa1476fad465'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:03.214130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:03.219597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 917/1857] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:02 2022] DELETE /identity/v3/users/fd36b0da8f0745e5b97eaa1476fad465 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:03.219934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:03.227116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.227413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.227694 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.270442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:03.273038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:03.289787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:03.326716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:03.392996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 941/1858] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:03.397496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.404664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 918/1859] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:03.425103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a3d92-e0da-4c1c-893b-b26faaaa3736 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.434242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.434954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.435241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.435516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.436259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:03.436586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.438241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.453602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 942/1860] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:03.458299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.467302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.469709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:03.477096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06313bc-5c94-4e8d-b730-1f01bf23f60a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.487445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.489094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.489578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.489820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.490956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.494484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.531853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 919/1861] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:03 2022] POST /identity/v3/projects => generated 375 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:03.535568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.574190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f532f276-fe90-49d2-a59d-f24e84516bff admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.640553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c03c732-a11b-4950-ba1a-262ecb924652 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.648259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.649390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.649751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.650097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.650876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.653155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.688511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 920/1862] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:03 2022] POST /identity/v3/users => generated 288 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:03.692395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.706646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.707531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.707531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.708170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.713940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39b47b3-860e-4471-a561-08b81bc54be7 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.721338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.722843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.722843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.722843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.723594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.725425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.750045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.752533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.770414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 921/1863] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:03.776967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.790527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 943/1864] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:03.806492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32feb8f-c604-4c74-a487-340fdeebe6a4 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.818169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57add9b4-38a8-48a4-bf08-e1c02aa17414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.822634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57add9b4-38a8-48a4-bf08-e1c02aa17414 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.823010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57add9b4-38a8-48a4-bf08-e1c02aa17414 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.823361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57add9b4-38a8-48a4-bf08-e1c02aa17414 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.824211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57add9b4-38a8-48a4-bf08-e1c02aa17414 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.829359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57add9b4-38a8-48a4-bf08-e1c02aa17414 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.850150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 922/1865] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:03.854237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:03.864353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9d048e-e3e3-439e-8aee-56b5d61df3e8 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:03.871980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:03.873340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.873670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.874111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] PATH_INFO: `/v3/projects/3fb5f614165a4c97acdf5c228a5dfd6e/users/843286b066f44a96820e78cbd0cbab46/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.892834 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fb5f614165a4c97acdf5c228a5dfd6e, user_id=843286b066f44a96820e78cbd0cbab46, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:03.895783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:03.916127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 944/1866] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:03 2022] PUT /identity/v3/projects/3fb5f614165a4c97acdf5c228a5dfd6e/users/843286b066f44a96820e78cbd0cbab46/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:03.927487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:03.929509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:03.930103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:03.968907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] MFA Rules not processed for user `843286b066f44a96820e78cbd0cbab46`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:03.970837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:04.052262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 923/1867] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:03 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:04.057144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.114331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:04.129714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3fe65f-e95c-4d10-a54c-e382a0098dd3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.135894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.137227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.137724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.138233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.139215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.141557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.185514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 945/1868] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:04 2022] POST /identity/v3/projects => generated 373 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:04.189565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.207585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe1ff5f-b686-4c53-a2d5-227dc693c72e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.216257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:04.298672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.299890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.299890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.300467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.301449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.304118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.335240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 924/1869] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:04 2022] POST /identity/v3/users => generated 293 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:04.339142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.351298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c3f6af-bd62-4588-99f2-3cedac329844 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.356339 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.357181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.357567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.357950 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.358784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.360515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.377338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 946/1870] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:04.378878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.395124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598cd981-7d03-4e8a-a55c-bdeeb3b83d25 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.403691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.404761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.405115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.405378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] PATH_INFO: `/v3/projects/a848644f965e4feeb7f5a29aea7af12b/users/80c2fc1db2054ccb973a90765764c937/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.426601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a848644f965e4feeb7f5a29aea7af12b, user_id=80c2fc1db2054ccb973a90765764c937, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.429861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.446643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 925/1871] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:04 2022] PUT /identity/v3/projects/a848644f965e4feeb7f5a29aea7af12b/users/80c2fc1db2054ccb973a90765764c937/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:04.450590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.463045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b4285-0014-410b-a594-150bcb5d5ac2 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.524633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.527066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:04.607924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.609366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.609961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.610507 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.614811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.623301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.650090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 947/1872] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:04 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:04.652775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.666158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c2784-ed71-44b1-83cf-d2cb6581ade8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.666972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebb7e4-c358-4eec-8877-0d5c957ea4de admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:04.673372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0781b5c1-5bde-4514-9e67-70466c0630a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.673550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:04.674343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0781b5c1-5bde-4514-9e67-70466c0630a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.674535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.674645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0781b5c1-5bde-4514-9e67-70466c0630a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.674822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.674924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0781b5c1-5bde-4514-9e67-70466c0630a5 admin admin] PATH_INFO: `/v3/projects/a848644f965e4feeb7f5a29aea7af12b/users/80c2fc1db2054ccb973a90765764c937/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.675074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] PATH_INFO: `/v3/projects/a36afd3997cb4202aa69a2de07790383` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.676243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a36afd3997cb4202aa69a2de07790383)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.677008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0781b5c1-5bde-4514-9e67-70466c0630a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a848644f965e4feeb7f5a29aea7af12b, user_id=80c2fc1db2054ccb973a90765764c937, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:04.679158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.679824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0781b5c1-5bde-4514-9e67-70466c0630a5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:04.702988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 948/1873] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:04 2022] PUT /identity/v3/projects/a848644f965e4feeb7f5a29aea7af12b/users/80c2fc1db2054ccb973a90765764c937/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:04.704953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.706590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.706965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.770409 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] MFA Rules not processed for user `80c2fc1db2054ccb973a90765764c937`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:04.772185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:04.789124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a36afd3997cb4202aa69a2de07790383', 'request_id': 'req-73110fbd-0e99-47ae-9bab-2061723e3b65'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:04.797440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a36afd3997cb4202aa69a2de07790383', 'request_id': 'req-73110fbd-0e99-47ae-9bab-2061723e3b65'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:04.809247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a36afd3997cb4202aa69a2de07790383', 'request_id': 'req-73110fbd-0e99-47ae-9bab-2061723e3b65'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:04.840200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] The token cache is being invalidate because project a36afd3997cb4202aa69a2de07790383 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:04.840505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:04.843212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-73110fbd-0e99-47ae-9bab-2061723e3b65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:04.848236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 926/1874] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:04 2022] DELETE /identity/v3/projects/a36afd3997cb4202aa69a2de07790383 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:04.854699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d4ee2-e077-4f08-b35e-3f8fda223ef4 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:04.855369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d4ee2-e077-4f08-b35e-3f8fda223ef4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:04.856720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d4ee2-e077-4f08-b35e-3f8fda223ef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:04.878845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:04.893136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 949/1875] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:04 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:04.900282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-029d4ee2-e077-4f08-b35e-3f8fda223ef4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:04.901795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-029d4ee2-e077-4f08-b35e-3f8fda223ef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:04.906064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:04.907734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:04.995012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 927/1876] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:05.004529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:05.014772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.016921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.017418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.042655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d113a-a418-47b0-a0c9-49d2a788155b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.047764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.048432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.048755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.049282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.052127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.057859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:05.061529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.076389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 928/1877] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:05.076389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.076939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.092590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5618098d-fb7b-47af-9128-e399475ee08a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.097999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.098855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.099134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.099401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] PATH_INFO: `/v3/projects/78d42b826c854caf90b655fa041914f1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.100632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78d42b826c854caf90b655fa041914f1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.102604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.140151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.141819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.154872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 950/1878] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:04 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:05.166838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.168652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78d42b826c854caf90b655fa041914f1', 'request_id': 'req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.181354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78d42b826c854caf90b655fa041914f1', 'request_id': 'req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.181719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1600a9-0dd9-4ad7-834d-dc753cf66b7e service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.195416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78d42b826c854caf90b655fa041914f1', 'request_id': 'req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.229222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] The token cache is being invalidate because project 78d42b826c854caf90b655fa041914f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:05.229518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.230371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.232424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 929/1879] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:05 2022] DELETE /identity/v3/projects/78d42b826c854caf90b655fa041914f1 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:05.236508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.238001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.273265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.274479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.274770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.275045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.275959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:05.276273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.278059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.295485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 951/1880] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:05.300790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.303634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.324330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c65779-f9d7-414a-bb84-cbbff66582a7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.332788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.333624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.334729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.335152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.335996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.338403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7862f914-0dc6-40d6-a7da-bc8045fcd2e1 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.351869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.353791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.353791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.353791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] PATH_INFO: `/v3/users/bbe0de9bcd3e4034ae5246a560059ac1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.356406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbe0de9bcd3e4034ae5246a560059ac1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.358703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.387855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 952/1881] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:05 2022] POST /identity/v3/projects => generated 375 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:05.389115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:05.398802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.422948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbe0de9bcd3e4034ae5246a560059ac1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.446655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbe0de9bcd3e4034ae5246a560059ac1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.461490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbe0de9bcd3e4034ae5246a560059ac1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.488159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbe0de9bcd3e4034ae5246a560059ac1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.489728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbe0de9bcd3e4034ae5246a560059ac1'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:05.525910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 930/1882] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:05 2022] DELETE /identity/v3/users/bbe0de9bcd3e4034ae5246a560059ac1 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:05.562421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7833c2df-244c-4754-9d7c-ff61efe55657 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.569085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:05.659859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.660573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.660899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.662988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.662988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.664736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.699476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 953/1883] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:05 2022] POST /identity/v3/users => generated 288 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:05.703964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.707095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.738910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688c19f4-4eaf-40bc-a31d-88121cbb35d9 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.754872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.755704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.756299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.756299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.757079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.758805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.782584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 931/1884] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:05 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:05.786437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.800435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218a638a-42fb-415c-ac5b-ea5d45562e7e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.806491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3635d5ee-9f0a-428b-a33c-f87fb2ad78a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.807380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3635d5ee-9f0a-428b-a33c-f87fb2ad78a8 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.807663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3635d5ee-9f0a-428b-a33c-f87fb2ad78a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.807938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3635d5ee-9f0a-428b-a33c-f87fb2ad78a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.808851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3635d5ee-9f0a-428b-a33c-f87fb2ad78a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.824554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3635d5ee-9f0a-428b-a33c-f87fb2ad78a8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.841914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 954/1885] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:05 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:05.843342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.855609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2415b03-fbba-4002-ab17-0239fdcb22ff admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:05.866396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:05.873111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.873609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.873895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] PATH_INFO: `/v3/projects/139e7e7d91c548beabab5f85ffad55d8/users/e714f7eb9e0c485ca30e0eb75d80fbc9/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.893063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] RBAC: Authorizing `identity:create_grant(project_id=139e7e7d91c548beabab5f85ffad55d8, user_id=e714f7eb9e0c485ca30e0eb75d80fbc9, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:05.896099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:05.914146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 932/1886] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:05 2022] PUT /identity/v3/projects/139e7e7d91c548beabab5f85ffad55d8/users/e714f7eb9e0c485ca30e0eb75d80fbc9/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:05.926049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:05.926674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:05.927175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:05.980242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:05.982037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:05.984604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] MFA Rules not processed for user `e714f7eb9e0c485ca30e0eb75d80fbc9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:05.985831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.062764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45649f5-fb87-44d1-abe6-b04f8de8274b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.066454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 955/1887] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:05 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:06.069201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.070192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.070565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.071177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] PATH_INFO: `/v3/users/8f188b7f8c7a4720b1aa3fdf8969c86d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.072614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f188b7f8c7a4720b1aa3fdf8969c86d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.074757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.079184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.080645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.138887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f188b7f8c7a4720b1aa3fdf8969c86d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.147641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f188b7f8c7a4720b1aa3fdf8969c86d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.150189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2592784-b3f7-4fb9-acca-75a99b4f42a0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.154055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.154769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.155117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.155390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.155496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f188b7f8c7a4720b1aa3fdf8969c86d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.159959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:06.164396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.166018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.170426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f188b7f8c7a4720b1aa3fdf8969c86d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.171536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b5a442a6-e361-4a3b-81d9-377ad3e99ebc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f188b7f8c7a4720b1aa3fdf8969c86d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.175558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 956/1888] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:06 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:06.176628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.178643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.180738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 933/1889] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:05 2022] DELETE /identity/v3/users/8f188b7f8c7a4720b1aa3fdf8969c86d => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:06.184292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.184462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.184764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.194334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7bf606-7fa9-4917-bff6-ab11fd22abde service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.203461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:06.204642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.211631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:06.259977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.261072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.261428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.261946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] PATH_INFO: `/v3/projects/b3ca13f3a22247c5a5ae89d9f16810b0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.263264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3ca13f3a22247c5a5ae89d9f16810b0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.265225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.272449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 934/1890] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:06.282709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.284914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.335442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3ca13f3a22247c5a5ae89d9f16810b0', 'request_id': 'req-e56dab4a-b6da-465a-a084-e6f25a6eef1a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.346005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3ca13f3a22247c5a5ae89d9f16810b0', 'request_id': 'req-e56dab4a-b6da-465a-a084-e6f25a6eef1a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.357123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3ca13f3a22247c5a5ae89d9f16810b0', 'request_id': 'req-e56dab4a-b6da-465a-a084-e6f25a6eef1a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.362472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1271aa-f779-421f-a9fa-7f0ca6472be0 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.373915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] The token cache is being invalidate because project b3ca13f3a22247c5a5ae89d9f16810b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:06.374240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.375470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e56dab4a-b6da-465a-a084-e6f25a6eef1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:06.377547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 957/1891] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:06 2022] DELETE /identity/v3/projects/b3ca13f3a22247c5a5ae89d9f16810b0 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:06.383481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc4f0ff-5bd6-4485-971f-b7e5986e3b83 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.384098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc4f0ff-5bd6-4485-971f-b7e5986e3b83 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.384533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc4f0ff-5bd6-4485-971f-b7e5986e3b83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.420045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9bc4f0ff-5bd6-4485-971f-b7e5986e3b83 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:06.422378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9bc4f0ff-5bd6-4485-971f-b7e5986e3b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.484491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.485185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.485471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.485745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.490101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.515697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 958/1892] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:06.523892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.524354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.524803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.561892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:06.566240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:06.606092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 959/1893] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:06.607880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.622159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=2,keystone:SELECT=98 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:06.639940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ee010e-1b4e-4807-a814-88cb30fb44df None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.643540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.649848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.659693 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.660380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.660664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.660962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.661713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:06.662021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.664316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.683616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 960/1894] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:06.684518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.689341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 935/1895] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:06.697091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb573b11-baa0-4850-8326-be68764f2c7c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.702050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.703302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.703302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.703800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.704758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.707849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.756016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 961/1896] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:06 2022] POST /identity/v3/projects => generated 377 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:06.761068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.773975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24245365-f133-4c32-aa22-bc4887367dc7 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.848741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.849441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.849783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.850066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.850775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.853161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.888083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 936/1897] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:06 2022] POST /identity/v3/users => generated 289 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:06.893451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.909529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b18bbaf-355e-4566-af58-3ff5aef85712 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.918187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.919417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.919940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.920396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.921396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:06.926649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:06.935681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:06.947683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 962/1898] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:06 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:06.951407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:06.967235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fced6a-0a93-48e1-a21a-9ebcf08448c0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:06.996625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c34e8e2-04df-4915-90ac-c36d3f41bb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:06.997560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c34e8e2-04df-4915-90ac-c36d3f41bb99 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:06.998112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c34e8e2-04df-4915-90ac-c36d3f41bb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:06.998404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c34e8e2-04df-4915-90ac-c36d3f41bb99 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:06.999341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c34e8e2-04df-4915-90ac-c36d3f41bb99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.001407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c34e8e2-04df-4915-90ac-c36d3f41bb99 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.035484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 937/1899] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:06 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:07.039862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.058577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad1fcd-9033-4c44-9f54-1d90ae02ee53 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:07.075861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:07.077137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.077597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.078018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] PATH_INFO: `/v3/projects/fe5c16794c9644bd93b9ae6b43223c59/users/8797d8fbd42c431f8daf7e4848a9e88f/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.119046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe5c16794c9644bd93b9ae6b43223c59, user_id=8797d8fbd42c431f8daf7e4848a9e88f, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.122090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.142955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 963/1900] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:07 2022] PUT /identity/v3/projects/fe5c16794c9644bd93b9ae6b43223c59/users/8797d8fbd42c431f8daf7e4848a9e88f/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:07.165208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.165591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.165856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.243654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] MFA Rules not processed for user `8797d8fbd42c431f8daf7e4848a9e88f`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:07.244581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:07.416451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 938/1901] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:07 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:07.421288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.513023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173956e2-8b1b-463f-a549-b90c8e943bbc admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:07.519099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:07.519775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.520258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.520536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.521260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.524163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.558909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 964/1902] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:07 2022] POST /identity/v3/projects => generated 375 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:07.562821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.575122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e64dc0-0be2-42b8-9cb0-bd632ab0e862 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:07.714453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:07.715362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.715927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.716856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.719502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.727394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.747796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:07.760823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.762707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:07.777544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 939/1903] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:07 2022] POST /identity/v3/users => generated 294 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:07.783777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46700c26-a4be-43e2-ae3e-2066e4c605e9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:07.793137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:07.794022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.794304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.794576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] PATH_INFO: `/v3/projects/43ac0fec8c6e4c50921ff8908093b725` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.796798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43ac0fec8c6e4c50921ff8908093b725)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.798740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4453e0-651b-4ee4-8ae9-ee04984f38b5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:07.800245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.806438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:07.807815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.808878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.810255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.810886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.813045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.830131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 940/1904] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:07 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:07.833079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.851553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf849c-0298-48a4-a34e-07911516b6f1 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:07.857245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:07.858226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:07.858594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:07.858821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] PATH_INFO: `/v3/projects/ee0dec6738b843bea4a5f087c3db1fc9/users/804dfbab161841db8bc585c13d57c4d3/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:07.878876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee0dec6738b843bea4a5f087c3db1fc9, user_id=804dfbab161841db8bc585c13d57c4d3, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:07.882714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:07.901746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 941/1905] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:07 2022] PUT /identity/v3/projects/ee0dec6738b843bea4a5f087c3db1fc9/users/804dfbab161841db8bc585c13d57c4d3/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:07.905412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:07.941170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43ac0fec8c6e4c50921ff8908093b725', 'request_id': 'req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:07.961284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43ac0fec8c6e4c50921ff8908093b725', 'request_id': 'req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:07.982577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43ac0fec8c6e4c50921ff8908093b725', 'request_id': 'req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:07.988327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77baed4d-85ff-45c1-9a67-2f8ca7bea736 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.000781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] The token cache is being invalidate because project 43ac0fec8c6e4c50921ff8908093b725 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:08.001145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.001988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.003921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 965/1906] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:07 2022] DELETE /identity/v3/projects/43ac0fec8c6e4c50921ff8908093b725 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:08.022915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:08.024600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.103431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.104126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.104457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.104762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.106438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.108576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.123051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 942/1907] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:07 2022] GET /identity/v3/roles => generated 1420 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:08.124106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:08.126039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:08.128500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.144451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d77ebc9-4d69-4146-81da-2abc19667cf0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.149481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.150369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.150660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.150937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] PATH_INFO: `/v3/users/843286b066f44a96820e78cbd0cbab46` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.152182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] RBAC: Authorizing `identity:delete_user(user_id=843286b066f44a96820e78cbd0cbab46)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.152726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f4a18-9e91-48ac-b1d0-7a0cb1e6f949 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.154146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.157886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.159159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.159497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.159725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] PATH_INFO: `/v3/users/a58460584e37489083803f5c026abb58` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.161748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a58460584e37489083803f5c026abb58)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.164247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.212582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '843286b066f44a96820e78cbd0cbab46'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.216892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:INSERT=5,keystone:SELECT=87,keystone:DELETE=8 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:08.222979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a58460584e37489083803f5c026abb58'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.226766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '843286b066f44a96820e78cbd0cbab46'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.232223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a58460584e37489083803f5c026abb58'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.236756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '843286b066f44a96820e78cbd0cbab46'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.241425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a58460584e37489083803f5c026abb58'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.242604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a58460584e37489083803f5c026abb58'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.243574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c2b1d4f-b551-4f71-b0c1-dab35f7c6a12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a58460584e37489083803f5c026abb58'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.253044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '843286b066f44a96820e78cbd0cbab46'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.254007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '843286b066f44a96820e78cbd0cbab46'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.261714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 943/1908] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:08 2022] DELETE /identity/v3/users/843286b066f44a96820e78cbd0cbab46 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:08.262515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:08.263999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.267452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 966/1909] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:08 2022] DELETE /identity/v3/users/a58460584e37489083803f5c026abb58 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:08.268717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.269270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.269376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.296135 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:08.299197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.392199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 967/1910] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:08.405475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36dd6c9-4ccc-4ee3-9c48-bccbc74a5039 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.408406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:08.411994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.415863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ff38bf-22d1-48c3-acab-5b71d9afb938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.416995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ff38bf-22d1-48c3-acab-5b71d9afb938 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.417414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ff38bf-22d1-48c3-acab-5b71d9afb938 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.417710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ff38bf-22d1-48c3-acab-5b71d9afb938 admin admin] PATH_INFO: `/v3/projects/ee0dec6738b843bea4a5f087c3db1fc9/users/804dfbab161841db8bc585c13d57c4d3/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.419933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ff38bf-22d1-48c3-acab-5b71d9afb938 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee0dec6738b843bea4a5f087c3db1fc9, user_id=804dfbab161841db8bc585c13d57c4d3, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.422843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ff38bf-22d1-48c3-acab-5b71d9afb938 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.435529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 944/1911] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:08 2022] PUT /identity/v3/projects/ee0dec6738b843bea4a5f087c3db1fc9/users/804dfbab161841db8bc585c13d57c4d3/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:08.445898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.446192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.446475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.487565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] MFA Rules not processed for user `804dfbab161841db8bc585c13d57c4d3`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:08.488991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.506842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868346f9-7322-40fc-80e8-b6de069175c0 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.522983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:08.586875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 945/1912] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:08 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:08.598444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:08.601294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.602393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.603332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.603836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.604316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.688254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:08.692535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5288cb80-94cf-4d60-af6b-1b8a3184df27 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.698028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.698762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.699123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.699477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.705547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.742241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.744266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.764620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 968/1913] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:08.765734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:08.769325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:08.791798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dffaba-f533-488d-ac57-dec5e7977dab service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:08.798763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:08.799784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.800211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.800498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] PATH_INFO: `/v3/projects/14795235b6a542e49556b922d872aa71` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.801525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.801848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=14795235b6a542e49556b922d872aa71)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:08.803441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc2b0f0-129a-4427-9807-b73519093da3 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.805309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:08.816793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 946/1914] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:08 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:08.883228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14795235b6a542e49556b922d872aa71', 'request_id': 'req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.889952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14795235b6a542e49556b922d872aa71', 'request_id': 'req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.899619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14795235b6a542e49556b922d872aa71', 'request_id': 'req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.922455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] The token cache is being invalidate because project 14795235b6a542e49556b922d872aa71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:08.923150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.924107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d1d7be2-0915-451c-9bb3-87d7e4c258ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:08.929968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 969/1915] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:08 2022] DELETE /identity/v3/projects/14795235b6a542e49556b922d872aa71 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:08.932404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:08.932810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:08.933640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:08.955813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:08.957140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.073798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 947/1916] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:09.076368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.076863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.077173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.099274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:09.100732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.127361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 970/1917] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:09.132654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.144390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e00bfe-e90b-4d0d-9661-2e765c089c4d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.150558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.151396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.152403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.152900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.154502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:09.154962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.158763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.180128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.180999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 948/1918] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:09.190333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2e8381-11af-4b41-86a5-4630e18e4958 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.201199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.202006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.202832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.202832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.203673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.206003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.240124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 971/1919] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:09 2022] POST /identity/v3/projects => generated 369 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:09.244894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.257294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668baf61-0e05-4eae-92ad-121a6c7e3315 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.273227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.275568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.321335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:09.328108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.328897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.329252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.329532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.330302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.332576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.358148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 949/1920] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:09 2022] POST /identity/v3/users => generated 285 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:09.361843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.372856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db2c19-062b-4907-8984-28022f36d6d7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.375295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd38022-97fe-47a1-9d5a-98ee36944b44 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.377711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.378580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.378866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.379142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] PATH_INFO: `/v3/projects/3fb5f614165a4c97acdf5c228a5dfd6e` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.380329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fb5f614165a4c97acdf5c228a5dfd6e)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.381434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.382089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.382374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.382653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.382806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.383374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.385829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.403812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 950/1921] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:09 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:09.405748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.417546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c20b6ac-dd49-434d-88d8-89b89a745a74 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.423352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.424025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.424305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.424738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.425362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.426992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.429645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:09.441412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 951/1922] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:09.445202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.446264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fb5f614165a4c97acdf5c228a5dfd6e', 'request_id': 'req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.453189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fb5f614165a4c97acdf5c228a5dfd6e', 'request_id': 'req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.456813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fd8871-6ff1-4001-b74a-c6a11d29770b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.462856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fb5f614165a4c97acdf5c228a5dfd6e', 'request_id': 'req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.475621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] The token cache is being invalidate because project 3fb5f614165a4c97acdf5c228a5dfd6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:09.476452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.477155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.479480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 972/1923] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:09 2022] DELETE /identity/v3/projects/3fb5f614165a4c97acdf5c228a5dfd6e => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:09.483229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.484547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.518971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e840cc-8f17-4e3d-9223-53699aaeed77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.519859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e840cc-8f17-4e3d-9223-53699aaeed77 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.520144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e840cc-8f17-4e3d-9223-53699aaeed77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.520420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e840cc-8f17-4e3d-9223-53699aaeed77 admin admin] PATH_INFO: `/v3/projects/ab114116402b47fc99df63841a3011b8/users/bd1466d77407449baf366b5298951a74/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.535249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e840cc-8f17-4e3d-9223-53699aaeed77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab114116402b47fc99df63841a3011b8, user_id=bd1466d77407449baf366b5298951a74, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.538127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e840cc-8f17-4e3d-9223-53699aaeed77 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.550414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 952/1924] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:09 2022] PUT /identity/v3/projects/ab114116402b47fc99df63841a3011b8/users/bd1466d77407449baf366b5298951a74/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:09.551605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7bfc16-b7be-4bee-8a9f-1b393abdc858 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.561776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.562002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.562283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.590507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] MFA Rules not processed for user `bd1466d77407449baf366b5298951a74`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:09.591734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.609037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.609911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.610207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.610481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] PATH_INFO: `/v3/users/80c2fc1db2054ccb973a90765764c937` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.611691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80c2fc1db2054ccb973a90765764c937)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.613641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.615123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:09.659510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 953/1925] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:09 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:09.664502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.665942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80c2fc1db2054ccb973a90765764c937'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.666089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.673603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80c2fc1db2054ccb973a90765764c937'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.679784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc75f0-8d4a-4a17-8bb5-a2f57e1dcc8b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.680718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80c2fc1db2054ccb973a90765764c937'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.681360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80c2fc1db2054ccb973a90765764c937'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.681997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80c2fc1db2054ccb973a90765764c937'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:09.684760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.685422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.685707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.685983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.686675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.688572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.700232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 973/1926] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:09 2022] DELETE /identity/v3/users/80c2fc1db2054ccb973a90765764c937 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:09.718585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 954/1927] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:09 2022] POST /identity/v3/projects => generated 367 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:09.719742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.721189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:09.787493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d675a0e1-3680-41e9-86cf-ba7149e5c7a1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.851481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.852169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.852579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.852901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.853783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.855635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.886424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 974/1928] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:09 2022] POST /identity/v3/users => generated 290 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:09.890093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.905576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed704be-0f15-47c8-a804-e035d87c832c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.911360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.912376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.912899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.913378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.914351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.916519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.931084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 955/1929] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:09 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:09.935044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:09.947179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08558513-4089-4b18-b21b-4f5447318f84 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:09.952341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:09.953383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:09.953770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:09.954148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] PATH_INFO: `/v3/projects/3380c5c27f494806aeca3e70552ff663/users/12240435239d4610a72c779fb1be26c5/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:09.970275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:09.975314 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=3380c5c27f494806aeca3e70552ff663, user_id=12240435239d4610a72c779fb1be26c5, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:09.979774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:09.998608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 975/1930] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:09 2022] PUT /identity/v3/projects/3380c5c27f494806aeca3e70552ff663/users/12240435239d4610a72c779fb1be26c5/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:10.002550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.013485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e1b199-5c23-4f2a-9f24-d1129ccfcb03 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.052034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:10.078146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.081096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.089770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.090974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.091480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.091785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.092840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.094947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.123517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 956/1931] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:10.128102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.141819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d202d0-ab15-427e-9e0c-2307b6a638b6 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.147285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af94bff-1b04-4793-8db5-a77683bdac5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.148297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af94bff-1b04-4793-8db5-a77683bdac5f admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.148655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af94bff-1b04-4793-8db5-a77683bdac5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.149033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af94bff-1b04-4793-8db5-a77683bdac5f admin admin] PATH_INFO: `/v3/projects/3380c5c27f494806aeca3e70552ff663/users/12240435239d4610a72c779fb1be26c5/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.151129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af94bff-1b04-4793-8db5-a77683bdac5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3380c5c27f494806aeca3e70552ff663, user_id=12240435239d4610a72c779fb1be26c5, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.153911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af94bff-1b04-4793-8db5-a77683bdac5f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.158307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fc32f0-3f3c-4862-8edc-f707bf67a6fd admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.165925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 957/1932] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:10 2022] PUT /identity/v3/projects/3380c5c27f494806aeca3e70552ff663/users/12240435239d4610a72c779fb1be26c5/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:10.176015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.176302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.176586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.207430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] MFA Rules not processed for user `12240435239d4610a72c779fb1be26c5`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:10.208670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.222435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.223318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.223604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.223888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] PATH_INFO: `/v3/projects/a848644f965e4feeb7f5a29aea7af12b` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.225692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=a848644f965e4feeb7f5a29aea7af12b)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.227626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.277520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a848644f965e4feeb7f5a29aea7af12b', 'request_id': 'req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:10.284086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a848644f965e4feeb7f5a29aea7af12b', 'request_id': 'req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:10.288724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 958/1933] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:10 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:10.297306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a848644f965e4feeb7f5a29aea7af12b', 'request_id': 'req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:10.300784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.302165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.309460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] The token cache is being invalidate because project a848644f965e4feeb7f5a29aea7af12b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:10.309785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:10.310448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd35ada5-85db-4ce2-8500-cc5b7ee2f9ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:10.315303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 976/1934] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:10 2022] DELETE /identity/v3/projects/a848644f965e4feeb7f5a29aea7af12b => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:10.316981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e632da8c-c6cb-4df2-89fd-7e44d4e36d2b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.317274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e632da8c-c6cb-4df2-89fd-7e44d4e36d2b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.317551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e632da8c-c6cb-4df2-89fd-7e44d4e36d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.339144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e632da8c-c6cb-4df2-89fd-7e44d4e36d2b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:10.339900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e632da8c-c6cb-4df2-89fd-7e44d4e36d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.368276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e475e-ce3a-4d81-94ef-471b35d56af7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.372060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:10.412681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 977/1935] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:10.420086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.420352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.420630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.431111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.431768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.432049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.432323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.434252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.442296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:10.443625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.449190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:10.452159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 978/1936] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:10.456023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.465125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58874c18-8be7-4d96-93f3-57c20d6f6cca None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.469755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.470393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.470684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.470960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.471676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:10.471980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.473615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.485726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 979/1937] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:10.490966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.502323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.502452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cf635-0111-4001-9168-1103fdf485fa admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.503754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.506979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.507619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.507902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.508173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.509153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.510761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.515840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 959/1938] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:10.543044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 980/1939] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:10 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:10.547525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.560998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d007808b-bc78-4851-b5f8-4a0fa90f1e83 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.564591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.566498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.626798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.627649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.627933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.628869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.629570 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.631706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.636701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce09706-a920-40d1-8eeb-cbaff372a567 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.641110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.642610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.642909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.643316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.646937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.655634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 960/1940] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:10 2022] POST /identity/v3/users => generated 287 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:10.659394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.674724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1eb8bd-4e9d-4768-8d5c-29f4f333e06b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.677192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.678369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.678656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.678933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.679659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.681285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.691376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 961/1941] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:10.695711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.706657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9997ac9-628b-48e3-978a-f0134f204814 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.714040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.714177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ee1646-0fa3-43f4-8322-963b869257ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.714768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ee1646-0fa3-43f4-8322-963b869257ec admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.715052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ee1646-0fa3-43f4-8322-963b869257ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.715326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ee1646-0fa3-43f4-8322-963b869257ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.715479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.717361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ee1646-0fa3-43f4-8322-963b869257ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.719130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ee1646-0fa3-43f4-8322-963b869257ec admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.724642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 981/1942] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:10.732722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 962/1943] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:10.734985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.746116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2553ab7a-213f-422f-8a3b-17f2263b72b9 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:10.750905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:10.751783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.752070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.752348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] PATH_INFO: `/v3/projects/88dc732812fc4fdd8e9c307f3565ba58/users/e2293054dd2740e781825cbd068d74f7/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.767509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] RBAC: Authorizing `identity:create_grant(project_id=88dc732812fc4fdd8e9c307f3565ba58, user_id=e2293054dd2740e781825cbd068d74f7, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:10.770194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:10.782995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 982/1944] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:10 2022] PUT /identity/v3/projects/88dc732812fc4fdd8e9c307f3565ba58/users/e2293054dd2740e781825cbd068d74f7/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:10.788239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3566fc-b783-450c-b12c-231605e4d59e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:10.788530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3566fc-b783-450c-b12c-231605e4d59e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:10.788877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3566fc-b783-450c-b12c-231605e4d59e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:10.806596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-af3566fc-b783-450c-b12c-231605e4d59e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:10.807827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-af3566fc-b783-450c-b12c-231605e4d59e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.869695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 963/1945] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:10.881522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:10.883711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:10.904599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:10.985760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec55415e-7182-4248-a402-9f1623715195 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:11.019968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a609084c-864d-4030-87a9-749d753d1fb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:11.020603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a609084c-864d-4030-87a9-749d753d1fb8 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:11.020922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a609084c-864d-4030-87a9-749d753d1fb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:11.021299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a609084c-864d-4030-87a9-749d753d1fb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:11.033530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a609084c-864d-4030-87a9-749d753d1fb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:11.035380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a609084c-864d-4030-87a9-749d753d1fb8 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:11.050247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 983/1946] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:11.805866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846bc34b-9290-4116-97b3-889604975079 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:11.806329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846bc34b-9290-4116-97b3-889604975079 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:11.806582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846bc34b-9290-4116-97b3-889604975079 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:11.808088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 964/1947] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:05:11.824872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c096b6e-70b4-4913-909f-4cee52e5f2a1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:11.825340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c096b6e-70b4-4913-909f-4cee52e5f2a1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:11.825340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c096b6e-70b4-4913-909f-4cee52e5f2a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:11.852278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9c096b6e-70b4-4913-909f-4cee52e5f2a1 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:11.854260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9c096b6e-70b4-4913-909f-4cee52e5f2a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:11.943802 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 984/1948] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:05:11 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:11.958510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846bc34b-9290-4116-97b3-889604975079 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:11.960806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-846bc34b-9290-4116-97b3-889604975079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:11.978655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846bc34b-9290-4116-97b3-889604975079 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:11.984347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:11.985046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:11.985320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:11.985846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:11.995574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:11.997611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:12.014569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 965/1949] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:11 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:12.470732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:12.471077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:12.471358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:12.519670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] MFA Rules not processed for user `e2293054dd2740e781825cbd068d74f7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:12.521102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:12.621091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 985/1950] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:12 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:12.630627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:12.642635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:12.737099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30c85e2-04ba-49c4-868e-59fcf3c179d7 service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:12.750479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4689c13-6e98-4f34-a353-5d50170c1ba7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:12.751589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4689c13-6e98-4f34-a353-5d50170c1ba7 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:12.752586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4689c13-6e98-4f34-a353-5d50170c1ba7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:12.753034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4689c13-6e98-4f34-a353-5d50170c1ba7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:12.776231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4689c13-6e98-4f34-a353-5d50170c1ba7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:12.778170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4689c13-6e98-4f34-a353-5d50170c1ba7 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:12.790381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 966/1951] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:12 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:14.446924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:14.448458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:14.461925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beca5e4-5f71-4810-8100-53059ea76a05 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:14.466764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:14.467647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:14.467932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:14.468209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] PATH_INFO: `/v3/users/bd1466d77407449baf366b5298951a74` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:14.469393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd1466d77407449baf366b5298951a74)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:14.471329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:14.513901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:14.525345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd1466d77407449baf366b5298951a74'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:14.533019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd1466d77407449baf366b5298951a74'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:14.540206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd1466d77407449baf366b5298951a74'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:14.540889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd1466d77407449baf366b5298951a74'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:14.541498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd1466d77407449baf366b5298951a74'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:14.560884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 986/1952] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:14 2022] DELETE /identity/v3/users/bd1466d77407449baf366b5298951a74 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:14.565423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:14.565866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:14.566003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:14.584638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:14.585947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:14.648899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 967/1953] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:14.658906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:14.660552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:14.720540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c261fbb-8e58-4398-9ac5-2f667bf6fdd1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:14.724217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:14.724898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:14.725186 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:14.725463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:14.734889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:14.736312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:14.746113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 987/1954] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:15.068622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:15.070217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:15.085394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076f44d1-b070-4d7c-9ec7-e9daa1b01213 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:15.091621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:15.092495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.092807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.093082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] PATH_INFO: `/v3/projects/ab114116402b47fc99df63841a3011b8` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.094276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab114116402b47fc99df63841a3011b8)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:15.096160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:15.152977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab114116402b47fc99df63841a3011b8', 'request_id': 'req-d3266326-e9a6-4a2e-8315-7be15ccdb950'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.159356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab114116402b47fc99df63841a3011b8', 'request_id': 'req-d3266326-e9a6-4a2e-8315-7be15ccdb950'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.167322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab114116402b47fc99df63841a3011b8', 'request_id': 'req-d3266326-e9a6-4a2e-8315-7be15ccdb950'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.183524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] The token cache is being invalidate because project ab114116402b47fc99df63841a3011b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:15.183821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.184478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.189087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 968/1955] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:15 2022] DELETE /identity/v3/projects/ab114116402b47fc99df63841a3011b8 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:15.190420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:15.200755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789941a3-872b-4acf-9c4b-bfa5f273755e service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:15.290158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:15.291055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.291343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.291622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] PATH_INFO: `/v3/users/12240435239d4610a72c779fb1be26c5` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.292891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12240435239d4610a72c779fb1be26c5)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:15.294829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:15.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12240435239d4610a72c779fb1be26c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.348163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12240435239d4610a72c779fb1be26c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.353072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:15.356845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12240435239d4610a72c779fb1be26c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.357752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12240435239d4610a72c779fb1be26c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.358493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-db8f9d2b-7137-4d5b-86e8-746c0d4e39c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12240435239d4610a72c779fb1be26c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.378504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 988/1956] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:15 2022] DELETE /identity/v3/users/12240435239d4610a72c779fb1be26c5 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:15.655424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:15.657236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:15.671525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3266326-e9a6-4a2e-8315-7be15ccdb950 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:15.703272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:15.734056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:15.734924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.735210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.735488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] PATH_INFO: `/v3/projects/3380c5c27f494806aeca3e70552ff663` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.736682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] RBAC: Authorizing `identity:delete_project(project_id=3380c5c27f494806aeca3e70552ff663)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:15.738597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:15.785335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3380c5c27f494806aeca3e70552ff663', 'request_id': 'req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.790744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3380c5c27f494806aeca3e70552ff663', 'request_id': 'req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.797952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3380c5c27f494806aeca3e70552ff663', 'request_id': 'req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.809565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] The token cache is being invalidate because project 3380c5c27f494806aeca3e70552ff663 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:15.809872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.810529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1066f6fe-2d84-4c29-83c0-9aceef0bc4de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:15.812069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 969/1957] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:15 2022] DELETE /identity/v3/projects/3380c5c27f494806aeca3e70552ff663 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:15.817055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.817541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.817633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.836232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:15.837517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:15.896684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 989/1958] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:15.904373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.904660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.904971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.922949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:15.924153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:15.932816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 970/1959] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:15.936799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:15.946604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d65cd8-a7d6-450d-bf2b-14c4bedfb6bc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:15.951221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:15.951863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.952147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.952444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.953223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:15.953524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:15.955108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:15.966936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 990/1960] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:15.971141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:15.981377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2050790-a0aa-4a03-87c6-e9e9f36d7735 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:15.986016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:15.986655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:15.986934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:15.987205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:15.987896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:15.989833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:16.015787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 971/1961] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:15 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:16.019875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:16.030375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156533d-54af-4221-920e-6daf8b1c51a7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:16.087348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:16.088017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:16.088319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:16.088576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:16.089340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:16.091272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:16.112404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 991/1962] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:16 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:16.116406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:16.127624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6566db-16d5-4bd4-aea2-96ea062efbba admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:16.132267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:16.132967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:16.133253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:16.133532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:16.134299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:16.135897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:16.149450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 972/1963] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:16.151043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:16.165364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9d2600-f2d3-4a60-9e3b-bd7efa1c68d3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:16.169966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d0e6d-36d9-4aec-92bd-e0e9302a5974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:16.170615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d0e6d-36d9-4aec-92bd-e0e9302a5974 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:16.170897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d0e6d-36d9-4aec-92bd-e0e9302a5974 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:16.171174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d0e6d-36d9-4aec-92bd-e0e9302a5974 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:16.171888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0d0e6d-36d9-4aec-92bd-e0e9302a5974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:16.173510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0d0e6d-36d9-4aec-92bd-e0e9302a5974 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:16.183848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 992/1964] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:16.187920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:16.197934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cf8ee1-02b1-4723-9f34-d7671c17e66c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:16.202525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:16.203394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:16.203680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:16.203953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] PATH_INFO: `/v3/projects/e5645cacf54248c18dd6ec70417a2b39/users/602af7cd7d4b466a939824889122ee2f/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:16.218186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5645cacf54248c18dd6ec70417a2b39, user_id=602af7cd7d4b466a939824889122ee2f, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:16.220824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:16.233652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 973/1965] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:16 2022] PUT /identity/v3/projects/e5645cacf54248c18dd6ec70417a2b39/users/602af7cd7d4b466a939824889122ee2f/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:16.238683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1f2786-3e27-4bfe-9f8f-47245eeb2385 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:16.238979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1f2786-3e27-4bfe-9f8f-47245eeb2385 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:16.239260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1f2786-3e27-4bfe-9f8f-47245eeb2385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:16.257805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1a1f2786-3e27-4bfe-9f8f-47245eeb2385 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:16.259015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1a1f2786-3e27-4bfe-9f8f-47245eeb2385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:16.300764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:16.328162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 993/1966] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:16.338283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:16.341489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:16.407443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3717a147-c5b9-4343-9e09-c1574feaef6e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:16.413503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:16.413848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:16.414768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:16.415108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:16.426551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:16.427989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:16.440068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 974/1967] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:17.371396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.372102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.372917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:17.399986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] MFA Rules not processed for user `602af7cd7d4b466a939824889122ee2f`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:17.401245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:17.469653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 994/1968] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:17 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:17.474081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.479829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:17.488187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45636146-8daf-4dba-8f9b-b5d383ea0ec8 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:17.493889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:17.494626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.494907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.495178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:17.496023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:17.498265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:17.535375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 975/1969] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:17 2022] POST /identity/v3/projects => generated 371 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:17.537101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.553037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80884342-e794-4cb2-8e0c-8b0505c0cb13 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:17.616378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:17.617637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.618085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.618555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:17.619847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:17.624123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:17.661276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.661695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 995/1970] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:17 2022] POST /identity/v3/users => generated 292 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:17.680396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714ca3c-aeaa-4c72-afe8-0b37d2fed637 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:17.686952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:17.688104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.688604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.689138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:17.689960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:17.691727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:17.707634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 976/1971] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:17 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:17.709176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.724068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affc157e-17cc-4105-80c6-851414b6744b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:17.737055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:17.737960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.738264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.738547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] PATH_INFO: `/v3/projects/472c78e143b6470ba727cedddbf45a64/users/89a9ad1f337f46939f7fc9885cff13f1/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:17.768600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] RBAC: Authorizing `identity:create_grant(project_id=472c78e143b6470ba727cedddbf45a64, user_id=89a9ad1f337f46939f7fc9885cff13f1, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:17.772353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:17.790940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 996/1972] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:17 2022] PUT /identity/v3/projects/472c78e143b6470ba727cedddbf45a64/users/89a9ad1f337f46939f7fc9885cff13f1/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:17.795834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.818761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16853358-eb2e-4b63-ab5e-6f6a7219a9d3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:17.926535 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:17.927558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.928068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.928539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:17.928955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.932735 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:17.933730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:17.936040 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:17.955261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 977/1973] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:17 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:17.959180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:17.976016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c8f608-4115-4339-a979-15127a514556 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:17.983111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7434ae-ad71-44c5-9bfa-46d0e5f3e434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:17.984513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7434ae-ad71-44c5-9bfa-46d0e5f3e434 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:17.985180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7434ae-ad71-44c5-9bfa-46d0e5f3e434 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:17.985652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7434ae-ad71-44c5-9bfa-46d0e5f3e434 admin admin] PATH_INFO: `/v3/projects/472c78e143b6470ba727cedddbf45a64/users/89a9ad1f337f46939f7fc9885cff13f1/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:18.000787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7434ae-ad71-44c5-9bfa-46d0e5f3e434 admin admin] RBAC: Authorizing `identity:create_grant(project_id=472c78e143b6470ba727cedddbf45a64, user_id=89a9ad1f337f46939f7fc9885cff13f1, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:18.004008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7434ae-ad71-44c5-9bfa-46d0e5f3e434 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:18.013860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:18.023896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 978/1974] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:17 2022] PUT /identity/v3/projects/472c78e143b6470ba727cedddbf45a64/users/89a9ad1f337f46939f7fc9885cff13f1/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:18.037629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50685c86-4456-4325-9654-f22848b86865 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:18.103373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:18.104654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:18.105244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:18.105696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:18.108180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:18.205033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:18.208246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4932a86d-c9c4-4c2c-88bd-910237c9356e service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:18.243036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 997/1975] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:17 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:18.458772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:18.459066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:18.459356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:18.508304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] MFA Rules not processed for user `e2293054dd2740e781825cbd068d74f7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:18.510682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:18.640509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 979/1976] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:18 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:19.232782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:19.233270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:19.234309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:19.270813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] MFA Rules not processed for user `89a9ad1f337f46939f7fc9885cff13f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:19.271800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:19.355024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 998/1977] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:19 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:19.368896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:19.371626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:19.387965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e34768-4e4c-46d3-b10b-c9eee0340b5e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:19.392613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:19.393772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:19.394583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:19.395058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:19.461613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:19.464040 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:19.466675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:19.475858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 980/1978] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:19 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:22.993354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:23.003202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32920c26-98cb-47f9-9394-958518cb5b67 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:23.006828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef9cb26-6c1d-4ac9-b92e-ca8522e2a63e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:23.007581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef9cb26-6c1d-4ac9-b92e-ca8522e2a63e service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:23.008023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef9cb26-6c1d-4ac9-b92e-ca8522e2a63e service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:23.008361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef9cb26-6c1d-4ac9-b92e-ca8522e2a63e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:23.021020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef9cb26-6c1d-4ac9-b92e-ca8522e2a63e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:23.022596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef9cb26-6c1d-4ac9-b92e-ca8522e2a63e service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:23.035990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 999/1979] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:22 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:29.472355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:33.028572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=93 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:34.089149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:34.090659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:34.161502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ea178-d35b-4f29-9a89-60e9ef5f931b service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:34.167310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:34.168674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:34.168851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:34.169031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] PATH_INFO: `/v3/users/e714f7eb9e0c485ca30e0eb75d80fbc9` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:34.170553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] RBAC: Authorizing `identity:delete_user(user_id=e714f7eb9e0c485ca30e0eb75d80fbc9)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:34.172510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:34.220072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e714f7eb9e0c485ca30e0eb75d80fbc9'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.227515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e714f7eb9e0c485ca30e0eb75d80fbc9'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.234304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e714f7eb9e0c485ca30e0eb75d80fbc9'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.246475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e714f7eb9e0c485ca30e0eb75d80fbc9'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.247368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e714f7eb9e0c485ca30e0eb75d80fbc9'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.253225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 981/1980] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:34 2022] DELETE /identity/v3/users/e714f7eb9e0c485ca30e0eb75d80fbc9 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:34.257761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:34.258209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:34.258310 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:34.275642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:34.276916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:34.334890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1000/1981] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:34.344552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:34.345999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:34.407882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f37f553-6d8c-420f-92ef-1e4c77ec0fba admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:34.411754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa4e7c-ac6a-45d1-8018-ffb177441936 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:34.412532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa4e7c-ac6a-45d1-8018-ffb177441936 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:34.412944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa4e7c-ac6a-45d1-8018-ffb177441936 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:34.413312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa4e7c-ac6a-45d1-8018-ffb177441936 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:34.424796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fa4e7c-ac6a-45d1-8018-ffb177441936 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:34.426477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fa4e7c-ac6a-45d1-8018-ffb177441936 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:34.437357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 982/1982] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:34.800663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:34.802126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:34.813858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306d97f9-080d-4329-98ed-1f7d98725d9f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:34.818647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:34.819508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:34.819794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:34.820660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] PATH_INFO: `/v3/projects/139e7e7d91c548beabab5f85ffad55d8` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:34.821871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=139e7e7d91c548beabab5f85ffad55d8)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:34.823779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:34.870653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '139e7e7d91c548beabab5f85ffad55d8', 'request_id': 'req-70c817c7-0424-458a-ae28-3f3be0d8c8fb'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.876829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '139e7e7d91c548beabab5f85ffad55d8', 'request_id': 'req-70c817c7-0424-458a-ae28-3f3be0d8c8fb'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.884951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '139e7e7d91c548beabab5f85ffad55d8', 'request_id': 'req-70c817c7-0424-458a-ae28-3f3be0d8c8fb'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.896257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] The token cache is being invalidate because project 139e7e7d91c548beabab5f85ffad55d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:34.896546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.897204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-70c817c7-0424-458a-ae28-3f3be0d8c8fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:34.898878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1001/1983] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:34 2022] DELETE /identity/v3/projects/139e7e7d91c548beabab5f85ffad55d8 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:34.903824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:34.904114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:34.904391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:34.922110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:34.923241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:34.941450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:34.992504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 983/1984] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:34.994388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:34.994675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:34.994953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.013864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:35.015059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:35.022937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1002/1985] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:35.027089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.036662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6a83be-a7af-4bc2-9fba-03b991452256 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.041764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.042411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.042697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.042970 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.043698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:35.044004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.045620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.057518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 984/1986] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:35.061531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.070509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ceda67-6569-4190-af3b-246d40c59fa9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.075113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.075754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.076039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.076315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.077056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.078963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.103987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1003/1987] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/projects => generated 385 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:35.108196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.118758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf6e832-088d-4e82-837e-706cf4714058 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.178579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.179165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.180396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.180396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.181125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.183039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.205805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 985/1988] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:35.207290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.218283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08801aa7-6551-424e-8d89-2f976774e00b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.225906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.227660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.228150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.228609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.229614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.231296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.242563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1004/1989] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:35.246518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.256849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0836ed-082a-478a-b79f-f59101663624 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.261438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a710164-04b2-4e44-9006-63a12f6b1c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.262109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a710164-04b2-4e44-9006-63a12f6b1c9a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.262392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a710164-04b2-4e44-9006-63a12f6b1c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.262670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a710164-04b2-4e44-9006-63a12f6b1c9a admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.263400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a710164-04b2-4e44-9006-63a12f6b1c9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.265034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a710164-04b2-4e44-9006-63a12f6b1c9a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.275981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 986/1990] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:35.280295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.290714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965ba73-a0e1-46cd-837e-ae05f78208ec admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.295303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.296192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.296463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.296773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] PATH_INFO: `/v3/projects/d034b1ea9fad4fd5a76a2cff5c661fa2/users/0d6eccd9d7a64900a6561a88988abe0b/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.311812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d034b1ea9fad4fd5a76a2cff5c661fa2, user_id=0d6eccd9d7a64900a6561a88988abe0b, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.314491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.329364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1005/1991] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:35 2022] PUT /identity/v3/projects/d034b1ea9fad4fd5a76a2cff5c661fa2/users/0d6eccd9d7a64900a6561a88988abe0b/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:35.341158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.341450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.341726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.369172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] MFA Rules not processed for user `0d6eccd9d7a64900a6561a88988abe0b`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:35.370472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:35.434464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 987/1992] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:35.438832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.459118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:35.504103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba063158-7c58-48b4-81d8-bd1bd4de2750 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.508795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.509454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.509739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.510012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.510700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.512624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.538487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1006/1993] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:35.543070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.553643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf3c264-ff4f-4e81-8990-11d0a604cd7a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.566135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:35.617157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.617831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.618119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.618395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.619094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.625826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.664097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 988/1994] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/users => generated 293 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:35.668546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.679571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a36da03-4d85-4148-8c56-7106d71a5022 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.684417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.685405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.685696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.686038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.686827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.688639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.705095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1007/1995] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:35.706977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.720413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904160bd-5f8f-4a97-b47f-6619f6de0287 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.725128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc40781a-7477-412e-93cf-3581fb994b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.726326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc40781a-7477-412e-93cf-3581fb994b35 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.726326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc40781a-7477-412e-93cf-3581fb994b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.726538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc40781a-7477-412e-93cf-3581fb994b35 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.727133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc40781a-7477-412e-93cf-3581fb994b35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.728733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc40781a-7477-412e-93cf-3581fb994b35 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.739971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 989/1996] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:35.744226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.754639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02c291d-4987-4edb-a67b-ba017e1c32a9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.760323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.761151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.761433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.761704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] PATH_INFO: `/v3/projects/53b55eae377c49a0966c18fed1072b6e/users/73a3f24126044c61bd65ca38b0acb205/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.778593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] RBAC: Authorizing `identity:create_grant(project_id=53b55eae377c49a0966c18fed1072b6e, user_id=73a3f24126044c61bd65ca38b0acb205, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.782019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:35.799378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1008/1997] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:35 2022] PUT /identity/v3/projects/53b55eae377c49a0966c18fed1072b6e/users/73a3f24126044c61bd65ca38b0acb205/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:35.815335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.815602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.815941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.845315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] MFA Rules not processed for user `73a3f24126044c61bd65ca38b0acb205`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:35.846517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:35.909991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 990/1998] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:35.914485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:35.975942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e618cea-6ac4-4ae7-bc13-191e8652f47f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:35.980820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:35.981484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:35.981825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:35.982108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:35.982805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:35.984731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.011583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1009/1999] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:35 2022] POST /identity/v3/projects => generated 385 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:36.015639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:36.026355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f0df70-1b19-4f14-a823-6b05ea1caf63 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:36.082219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:36.082889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.083173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.083448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.084148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:36.086120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.108156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 991/2000] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:36 2022] POST /identity/v3/users => generated 299 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:36.109440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:36.120300 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3374d56-7fae-40dc-8b44-c6354ebb2e55 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:36.125217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:36.125871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.126162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.126441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.127174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:36.128785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.139769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1010/2001] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:36.143794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:36.153523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33622f-7971-4e74-9f49-c00aedeecddf admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:36.158102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:36.158969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.159251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.159526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] PATH_INFO: `/v3/projects/16486734993e43529fb051860e6e8fb4/users/b7deb7b3e3ec4486be593bc56fe4eef4/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.172119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:36.175519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16486734993e43529fb051860e6e8fb4, user_id=b7deb7b3e3ec4486be593bc56fe4eef4, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:36.178922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.192818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 992/2002] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:36 2022] PUT /identity/v3/projects/16486734993e43529fb051860e6e8fb4/users/b7deb7b3e3ec4486be593bc56fe4eef4/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:36.196792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:36.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6980cf-2971-4b25-a646-7bf3051577e1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:36.250022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:36.265959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d2b683-3c42-4cc1-8545-2bc68645c4a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:36.266617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d2b683-3c42-4cc1-8545-2bc68645c4a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.266898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d2b683-3c42-4cc1-8545-2bc68645c4a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.267168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d2b683-3c42-4cc1-8545-2bc68645c4a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.267884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d2b683-3c42-4cc1-8545-2bc68645c4a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:36.269492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d2b683-3c42-4cc1-8545-2bc68645c4a9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.279688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1011/2003] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:36 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:36.283869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:36.294517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccd569-df84-428a-b857-8afd147a3671 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:36.299148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:36.300023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.300315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.300588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] PATH_INFO: `/v3/projects/16486734993e43529fb051860e6e8fb4/users/b7deb7b3e3ec4486be593bc56fe4eef4/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.302680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16486734993e43529fb051860e6e8fb4, user_id=b7deb7b3e3ec4486be593bc56fe4eef4, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:36.305410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.315508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 993/2004] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:36 2022] PUT /identity/v3/projects/16486734993e43529fb051860e6e8fb4/users/b7deb7b3e3ec4486be593bc56fe4eef4/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:36.326076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.326573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.326661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.353896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] MFA Rules not processed for user `b7deb7b3e3ec4486be593bc56fe4eef4`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:36.355142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:36.428475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1012/2005] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:36 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:36.440027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:36.441870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:36.506504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d002f7a-df99-4219-bd52-bbeeb8a3d381 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:36.510800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:36.511452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:36.511738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:36.512022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:36.579195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:36.581316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:36.593820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 994/2006] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:36 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:38.270474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:38.283295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd821b4c-068c-40aa-a410-3f10e68c7681 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:38.294134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:38.295089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:38.295467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:38.295907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:38.372819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:38.374559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:38.387436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1013/2007] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:38 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:38.992157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:39.000874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8387a739-bb82-461b-9628-9d5c65203729 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:39.004503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3244cd31-e1a0-4dcd-a19e-65fa6d027fda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:39.005180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3244cd31-e1a0-4dcd-a19e-65fa6d027fda service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:39.005466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3244cd31-e1a0-4dcd-a19e-65fa6d027fda service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:39.005739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3244cd31-e1a0-4dcd-a19e-65fa6d027fda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:39.014780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3244cd31-e1a0-4dcd-a19e-65fa6d027fda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:39.016249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3244cd31-e1a0-4dcd-a19e-65fa6d027fda service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:39.026582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 995/2008] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:38 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:46.207349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:46.275311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45da5f42-88e8-41d7-a761-ac601c0cab63 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:46.280217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:46.281144 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:46.281431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:46.281711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] PATH_INFO: `/v3/users/0d6eccd9d7a64900a6561a88988abe0b` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:46.282944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d6eccd9d7a64900a6561a88988abe0b)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:46.284939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:46.291851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:46.342676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d6eccd9d7a64900a6561a88988abe0b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:46.351365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d6eccd9d7a64900a6561a88988abe0b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:46.363245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d6eccd9d7a64900a6561a88988abe0b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:46.364429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d6eccd9d7a64900a6561a88988abe0b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:46.365413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0bc900a-f6d3-4119-9ba3-df5d372b4032 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d6eccd9d7a64900a6561a88988abe0b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:46.393501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1014/2009] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:46 2022] DELETE /identity/v3/users/0d6eccd9d7a64900a6561a88988abe0b => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:46.399100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:46.399669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:46.399669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:46.417879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:46.425308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:46.499800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 996/2010] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:46.535621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649aa129-2126-445c-90c9-64bb3bdca35c None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:46.536250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649aa129-2126-445c-90c9-64bb3bdca35c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:46.536790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649aa129-2126-445c-90c9-64bb3bdca35c None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:46.539032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1015/2011] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:05:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:05:46.547853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:46.549857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:46.575406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:46.575778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:46.576127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:46.583726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:46.601618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:46.602753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:46.679732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca14647-8243-445c-8055-0c54fe9006b4 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:46.689005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:46.689626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:46.689912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:46.690185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:46.709306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1016/2012] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:05:46 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:46.712724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:46.714581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:46.726872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 997/2013] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:46.738690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:46.740146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:46.759118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e1de83-faf9-4eaf-8fb3-4f3677625117 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:46.766659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a07c3c1-72f0-4258-8fed-2e67eca2454d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:46.767913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a07c3c1-72f0-4258-8fed-2e67eca2454d service placement] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:46.768419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a07c3c1-72f0-4258-8fed-2e67eca2454d service placement] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:46.768940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a07c3c1-72f0-4258-8fed-2e67eca2454d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:46.781051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a07c3c1-72f0-4258-8fed-2e67eca2454d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:46.783248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a07c3c1-72f0-4258-8fed-2e67eca2454d service placement] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:46.810244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1017/2014] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:46 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:46.981211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:46.983131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:47.003312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8097f874-a138-4009-8a1f-bbe6e5b768f2 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:47.009736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:47.010621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:47.010923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:47.011179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] PATH_INFO: `/v3/users/8797d8fbd42c431f8daf7e4848a9e88f` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:47.012381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8797d8fbd42c431f8daf7e4848a9e88f)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:47.014409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:47.117917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8797d8fbd42c431f8daf7e4848a9e88f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.138388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8797d8fbd42c431f8daf7e4848a9e88f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.149659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8797d8fbd42c431f8daf7e4848a9e88f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.169173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8797d8fbd42c431f8daf7e4848a9e88f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.170078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8797d8fbd42c431f8daf7e4848a9e88f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.181947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 998/2015] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:46 2022] DELETE /identity/v3/users/8797d8fbd42c431f8daf7e4848a9e88f => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:47.186429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:47.186748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:47.187008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:47.207128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:47.208457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:47.294921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1018/2016] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:47.350759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:47.352852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:47.417305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79804388-ac4f-407d-aff7-968d77c733d9 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:47.422335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:47.423522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:47.424017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:47.424499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:47.437444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:47.447294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:47.461458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 999/2017] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:47.549380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:47.551082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:47.563371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bce3b3-4a8a-472b-a874-49fa251460be None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:47.568148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:47.569313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:47.570344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:47.571092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] PATH_INFO: `/v3/projects/d034b1ea9fad4fd5a76a2cff5c661fa2` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:47.572644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d034b1ea9fad4fd5a76a2cff5c661fa2)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:47.574584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:47.576668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:47.650746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd034b1ea9fad4fd5a76a2cff5c661fa2', 'request_id': 'req-9c21f7b7-5a5e-4af5-8db3-7985642e5938'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.658174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd034b1ea9fad4fd5a76a2cff5c661fa2', 'request_id': 'req-9c21f7b7-5a5e-4af5-8db3-7985642e5938'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.666527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd034b1ea9fad4fd5a76a2cff5c661fa2', 'request_id': 'req-9c21f7b7-5a5e-4af5-8db3-7985642e5938'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.680593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] The token cache is being invalidate because project d034b1ea9fad4fd5a76a2cff5c661fa2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:47.680915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.681576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.683142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1019/2018] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:47 2022] DELETE /identity/v3/projects/d034b1ea9fad4fd5a76a2cff5c661fa2 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:47.687159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:47.699973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c8b1d-39bd-4eb7-90e3-f68719efaf60 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:47.744986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:47.775716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:47.776763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:47.777110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:47.777457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] PATH_INFO: `/v3/users/73a3f24126044c61bd65ca38b0acb205` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:47.778756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=73a3f24126044c61bd65ca38b0acb205)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:47.780825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:47.849158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:47.850703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:47.854557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73a3f24126044c61bd65ca38b0acb205'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.865084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73a3f24126044c61bd65ca38b0acb205'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.874139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c21f7b7-5a5e-4af5-8db3-7985642e5938 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:47.880084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:47.880940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:47.881246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:47.882067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] PATH_INFO: `/v3/users/602af7cd7d4b466a939824889122ee2f` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:47.884039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=602af7cd7d4b466a939824889122ee2f)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:47.887705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:47.896719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73a3f24126044c61bd65ca38b0acb205'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.916665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73a3f24126044c61bd65ca38b0acb205'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.917651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73a3f24126044c61bd65ca38b0acb205'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.924582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1000/2019] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:47 2022] DELETE /identity/v3/users/73a3f24126044c61bd65ca38b0acb205 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:47.962697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '602af7cd7d4b466a939824889122ee2f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.980415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '602af7cd7d4b466a939824889122ee2f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.991818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '602af7cd7d4b466a939824889122ee2f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:47.999453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '602af7cd7d4b466a939824889122ee2f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.001553 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '602af7cd7d4b466a939824889122ee2f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.033839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:48.034008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1020/2020] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:47 2022] DELETE /identity/v3/users/602af7cd7d4b466a939824889122ee2f => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:48.035963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:48.121176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f044a4c-3ab8-4f09-b851-d3ed889ee0bb admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:48.200239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:48.201949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:48.201949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:48.202258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] PATH_INFO: `/v3/projects/fe5c16794c9644bd93b9ae6b43223c59` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:48.203481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe5c16794c9644bd93b9ae6b43223c59)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:48.205730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:48.218384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:48.280596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe5c16794c9644bd93b9ae6b43223c59', 'request_id': 'req-26a7f46a-bc63-4335-b237-6608ae6114e2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.312426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe5c16794c9644bd93b9ae6b43223c59', 'request_id': 'req-26a7f46a-bc63-4335-b237-6608ae6114e2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.322327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:48.324106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:48.328736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe5c16794c9644bd93b9ae6b43223c59', 'request_id': 'req-26a7f46a-bc63-4335-b237-6608ae6114e2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.356282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] The token cache is being invalidate because project fe5c16794c9644bd93b9ae6b43223c59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:48.356720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.359512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e591b9cd-ffe3-4dd7-99c6-776320d9fd40 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:48.360544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.368060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1001/2021] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:48 2022] DELETE /identity/v3/projects/fe5c16794c9644bd93b9ae6b43223c59 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:48.369600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:48.371229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:48.466713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:48.467590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:48.467877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:48.468157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] PATH_INFO: `/v3/projects/e5645cacf54248c18dd6ec70417a2b39` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:48.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5645cacf54248c18dd6ec70417a2b39)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:48.487571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a7f46a-bc63-4335-b237-6608ae6114e2 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:48.489752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:48.495754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:48.496722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:48.497267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:48.497559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] PATH_INFO: `/v3/users/804dfbab161841db8bc585c13d57c4d3` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:48.500035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=804dfbab161841db8bc585c13d57c4d3)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:48.501749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:48.506927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:48.586234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '804dfbab161841db8bc585c13d57c4d3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.597004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '804dfbab161841db8bc585c13d57c4d3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.603800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5645cacf54248c18dd6ec70417a2b39', 'request_id': 'req-7110d6b3-1324-4af0-ac11-f724c4eef152'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.609389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '804dfbab161841db8bc585c13d57c4d3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.610455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5645cacf54248c18dd6ec70417a2b39', 'request_id': 'req-7110d6b3-1324-4af0-ac11-f724c4eef152'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.621246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5645cacf54248c18dd6ec70417a2b39', 'request_id': 'req-7110d6b3-1324-4af0-ac11-f724c4eef152'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.629242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '804dfbab161841db8bc585c13d57c4d3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.629242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '804dfbab161841db8bc585c13d57c4d3'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.634988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1002/2022] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:48 2022] DELETE /identity/v3/users/804dfbab161841db8bc585c13d57c4d3 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:48.635969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:48.637333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:48.638316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] The token cache is being invalidate because project e5645cacf54248c18dd6ec70417a2b39 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:48.638620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.639460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.642064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1021/2023] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:48 2022] DELETE /identity/v3/projects/e5645cacf54248c18dd6ec70417a2b39 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:48.647967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:48.651240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:48.654140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:48.749079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdb8849-69be-4dbb-ac39-177156bcaa92 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:48.757011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:48.759395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:48.759625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:48.760311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] PATH_INFO: `/v3/projects/53b55eae377c49a0966c18fed1072b6e` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:48.762391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] RBAC: Authorizing `identity:delete_project(project_id=53b55eae377c49a0966c18fed1072b6e)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:48.764888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:48.784629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7110d6b3-1324-4af0-ac11-f724c4eef152 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:48.791826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:48.792814 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:48.793087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:48.793412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] PATH_INFO: `/v3/users/89a9ad1f337f46939f7fc9885cff13f1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:48.794986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89a9ad1f337f46939f7fc9885cff13f1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:48.802393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:48.834554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53b55eae377c49a0966c18fed1072b6e', 'request_id': 'req-de763e96-ff51-4dc2-a5de-d850871a3dcd'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.842263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53b55eae377c49a0966c18fed1072b6e', 'request_id': 'req-de763e96-ff51-4dc2-a5de-d850871a3dcd'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.854543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53b55eae377c49a0966c18fed1072b6e', 'request_id': 'req-de763e96-ff51-4dc2-a5de-d850871a3dcd'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.882966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] The token cache is being invalidate because project 53b55eae377c49a0966c18fed1072b6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:48.883136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.886365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.890634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1003/2024] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:48 2022] DELETE /identity/v3/projects/53b55eae377c49a0966c18fed1072b6e => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:48.895184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:48.898325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:48.899882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89a9ad1f337f46939f7fc9885cff13f1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.909794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89a9ad1f337f46939f7fc9885cff13f1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.919463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89a9ad1f337f46939f7fc9885cff13f1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.920570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89a9ad1f337f46939f7fc9885cff13f1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.921472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89a9ad1f337f46939f7fc9885cff13f1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:48.953722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1022/2025] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:48 2022] DELETE /identity/v3/users/89a9ad1f337f46939f7fc9885cff13f1 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:49.025471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de763e96-ff51-4dc2-a5de-d850871a3dcd admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.072663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.074345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.089295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:49.120228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:49.135949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.138978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.139384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.139773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] PATH_INFO: `/v3/users/b7deb7b3e3ec4486be593bc56fe4eef4` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.141287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7deb7b3e3ec4486be593bc56fe4eef4)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.143367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.171309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5834fe-a0e3-4ff2-ba6e-307d72b6ee87 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.177874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.180041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.180445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.180919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] PATH_INFO: `/v3/projects/ee0dec6738b843bea4a5f087c3db1fc9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.182567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee0dec6738b843bea4a5f087c3db1fc9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.184634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.200977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7deb7b3e3ec4486be593bc56fe4eef4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.209905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7deb7b3e3ec4486be593bc56fe4eef4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.218976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7deb7b3e3ec4486be593bc56fe4eef4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.235337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7deb7b3e3ec4486be593bc56fe4eef4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.236437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b0a0a5b7-f619-49ab-a111-966ed9019079 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7deb7b3e3ec4486be593bc56fe4eef4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.242667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee0dec6738b843bea4a5f087c3db1fc9', 'request_id': 'req-d4a9282a-6750-4803-a2da-9925a222494f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.243992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1004/2026] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:48 2022] DELETE /identity/v3/users/b7deb7b3e3ec4486be593bc56fe4eef4 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:49.252503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee0dec6738b843bea4a5f087c3db1fc9', 'request_id': 'req-d4a9282a-6750-4803-a2da-9925a222494f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.262440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee0dec6738b843bea4a5f087c3db1fc9', 'request_id': 'req-d4a9282a-6750-4803-a2da-9925a222494f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.279692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] The token cache is being invalidate because project ee0dec6738b843bea4a5f087c3db1fc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:49.279993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.280805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d4a9282a-6750-4803-a2da-9925a222494f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.286147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1023/2027] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:49 2022] DELETE /identity/v3/projects/ee0dec6738b843bea4a5f087c3db1fc9 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:49.290024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.290378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.290679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.322986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:49.324269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.400913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1005/2028] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:49.412252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.412795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.412891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.445919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:49.447279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.456072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1024/2029] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:49.461844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.473658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d4635-0338-4853-936a-d0bd892839ae None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.479239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.480004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.480363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.480928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.481745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:49.482123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.483808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.496777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1006/2030] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:49.501095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.511900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0ae7e2-5481-4ecb-bba9-8460ee21e814 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.517138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.517963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.518376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.518721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.519519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.521930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.543751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.546130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.565544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac152a15-8280-4e7f-a5d3-0df5484faa43 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.582102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1025/2031] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/projects => generated 387 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:49.588193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.589743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.591064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.593178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.593607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] PATH_INFO: `/v3/projects/472c78e143b6470ba727cedddbf45a64` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.594657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.595740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] RBAC: Authorizing `identity:delete_project(project_id=472c78e143b6470ba727cedddbf45a64)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.599429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.682421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '472c78e143b6470ba727cedddbf45a64', 'request_id': 'req-65394a80-a625-4de6-b6a4-e1bcd2d57278'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.684893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:49.687525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9deed-eafa-444f-a5b0-9cee942bec11 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.696351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '472c78e143b6470ba727cedddbf45a64', 'request_id': 'req-65394a80-a625-4de6-b6a4-e1bcd2d57278'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.711591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '472c78e143b6470ba727cedddbf45a64', 'request_id': 'req-65394a80-a625-4de6-b6a4-e1bcd2d57278'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.712690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:49.729022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] The token cache is being invalidate because project 472c78e143b6470ba727cedddbf45a64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:49.729764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.733743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.736170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1007/2032] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:49 2022] DELETE /identity/v3/projects/472c78e143b6470ba727cedddbf45a64 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:49.737240 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.738937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.768616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.769516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.769801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.770076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] PATH_INFO: `/v3/projects/16486734993e43529fb051860e6e8fb4` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.771252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16486734993e43529fb051860e6e8fb4)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.773316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.823907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65394a80-a625-4de6-b6a4-e1bcd2d57278 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.824594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16486734993e43529fb051860e6e8fb4', 'request_id': 'req-a65cc7e8-ad7b-438e-b16e-c24706597b43'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.830149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16486734993e43529fb051860e6e8fb4', 'request_id': 'req-a65cc7e8-ad7b-438e-b16e-c24706597b43'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.837991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16486734993e43529fb051860e6e8fb4', 'request_id': 'req-a65cc7e8-ad7b-438e-b16e-c24706597b43'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.849638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] The token cache is being invalidate because project 16486734993e43529fb051860e6e8fb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:49.849923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.850601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a65cc7e8-ad7b-438e-b16e-c24706597b43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:49.852686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1026/2033] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:49 2022] DELETE /identity/v3/projects/16486734993e43529fb051860e6e8fb4 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:49.853933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.854220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.854496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.873237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:49.874452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.885270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaee668-4b4e-418d-a1e2-394217829fee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.886085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaee668-4b4e-418d-a1e2-394217829fee admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.886368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaee668-4b4e-418d-a1e2-394217829fee admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.886644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaee668-4b4e-418d-a1e2-394217829fee admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.887335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaee668-4b4e-418d-a1e2-394217829fee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.889332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaee668-4b4e-418d-a1e2-394217829fee admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.909457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1008/2034] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/users => generated 294 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:49.910719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07581b1-ae0b-4505-971c-77ec4fc6fc5a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.911008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07581b1-ae0b-4505-971c-77ec4fc6fc5a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.911284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07581b1-ae0b-4505-971c-77ec4fc6fc5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.935530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d07581b1-ae0b-4505-971c-77ec4fc6fc5a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:49.941864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1027/2035] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:49.942562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d07581b1-ae0b-4505-971c-77ec4fc6fc5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.942658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.942658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.950885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1009/2036] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:49.951913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f953af16-2be7-4b3f-9fde-f38d13ecaef4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.952130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.952415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.952687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.962282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba765ec4-7d66-482a-9d65-db23c206e7a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.962945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba765ec4-7d66-482a-9d65-db23c206e7a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.963230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba765ec4-7d66-482a-9d65-db23c206e7a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.963509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba765ec4-7d66-482a-9d65-db23c206e7a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.964308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba765ec4-7d66-482a-9d65-db23c206e7a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:49.966019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba765ec4-7d66-482a-9d65-db23c206e7a1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:49.973525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:49.974728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:49.976281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1028/2037] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:49 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:49.977573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.977891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.978177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.983035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1010/2038] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:49.983687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:49.993090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a7aebb-d25c-4abf-8000-0752e9324335 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:49.998152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:49.998437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:49.998783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:49.999141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:49.999418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:49.999620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:50.000272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.001880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.008117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1029/2039] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:50.011594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.017713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1011/2040] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.017869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac8e24d-7ccd-4558-ab75-5fc0d96d3056 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.018774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.022327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.023598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.023901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.024177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.024925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:50.025229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.026816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.029269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c3860-3d3a-43e0-97b0-b3840bc0ae7e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.034061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.034696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.034986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.035267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.035984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:50.036298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.038374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1030/2041] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:50.038757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.039153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.049766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c040d4-1bc3-4597-a5c4-5f1c4950b251 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.054785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1012/2042] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:50.054785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.058653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.059532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.059899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.060205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] PATH_INFO: `/v3/projects/04c52dd23a8f414389f5e0171da297cc/users/0a3309df0a7848228c82f154224980a0/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.066217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fe5d47-6f92-4aa5-8541-d52b24b50b0d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.074891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9c448c-77ef-4582-8eb3-336aa0a15acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.075676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9c448c-77ef-4582-8eb3-336aa0a15acd admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.076031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9c448c-77ef-4582-8eb3-336aa0a15acd admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.076381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9c448c-77ef-4582-8eb3-336aa0a15acd admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.077202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9c448c-77ef-4582-8eb3-336aa0a15acd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.079350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] RBAC: Authorizing `identity:create_grant(project_id=04c52dd23a8f414389f5e0171da297cc, user_id=0a3309df0a7848228c82f154224980a0, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.079491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9c448c-77ef-4582-8eb3-336aa0a15acd admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.082296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.087577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:50.103480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1031/2043] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:50 2022] PUT /identity/v3/projects/04c52dd23a8f414389f5e0171da297cc/users/0a3309df0a7848228c82f154224980a0/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:50.117118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1013/2044] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/projects => generated 361 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:50.117347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.117455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.117455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.117455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.155872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] MFA Rules not processed for user `0a3309df0a7848228c82f154224980a0`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:50.157153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:50.163312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=93,keystone:DELETE=2,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:50.219584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdf520-85e7-44e9-be4f-52006847720a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.251008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1014/2045] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:50.252143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.297754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.298515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.298870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.299216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.299995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.302069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.321904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40c9bdd-1e3b-4838-bd86-718adb2180a8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.327354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.328265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.328757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.329254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.330191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.332368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.335445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1032/2046] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/projects => generated 363 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:50.336415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.355833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1015/2047] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/users => generated 281 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:50.357342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ca2d8d-ccc8-40b4-9008-f7104b2a1d37 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.411133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed74f0fb-cea9-4edf-91b3-ba2f1c8f8e62 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.416082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.416806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.417200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.417490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.418211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.420107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.427449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.428182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.428468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.428770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.429996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.431894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.447969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1033/2048] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/projects => generated 385 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:50.448763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.453338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1016/2049] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/users => generated 282 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:50.454159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.462309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:50.464255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28412f0-392f-45c6-8803-076c70349768 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.489727 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:50.514008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8216a79-dd60-40de-8f3d-d1426f102253 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.518794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.519465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.519757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.520034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.520790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.522501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.524211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.524911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.525197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.525470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.526164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.528109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.532966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1034/2050] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.533908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.543538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d62ca-d64d-4e02-beb6-82009c5bc2ad admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.548488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.549158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.549442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.549736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.550429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.555178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1017/2051] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:50.555311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.555401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.562642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37751071-09f0-4fd4-a8ae-7667265eb944 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.567299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1035/2052] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.567299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.567523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.568149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.568434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.568845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.569452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.571102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.574671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd7fd62-0343-4e45-bec0-4183194dbf19 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.579434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.580070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.580357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.580637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.584102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1018/2053] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.584231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.584339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.584426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.591504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd899b8-8604-4d27-90bf-0cbbd960a161 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.596521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1036/2054] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.596521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.596721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.597174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.597458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.597732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.598558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.600197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.604642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b5ded4-a6e9-494a-9b5c-367b0ccd7b2d admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.609358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.610227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.610517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.610819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] PATH_INFO: `/v3/projects/8d0e1fda79e3489685753c227d79c8f2/users/e1356bcc15364d4fa3a8b54ba2105b88/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.611539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1019/2055] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.612306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.623659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56686d11-d953-42f7-a444-babd97af34ba admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.628179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.628830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.629114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.629386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.630093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.630957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d0e1fda79e3489685753c227d79c8f2, user_id=e1356bcc15364d4fa3a8b54ba2105b88, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.631651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.634986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.645488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1020/2056] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:50.645488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.649938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1037/2057] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:50 2022] PUT /identity/v3/projects/8d0e1fda79e3489685753c227d79c8f2/users/e1356bcc15364d4fa3a8b54ba2105b88/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:50.650793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.653862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e29297-c825-4db1-acf4-7a0918b2ecb1 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.710574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c15a9-4f5e-491a-b20d-335965f66e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.711571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c15a9-4f5e-491a-b20d-335965f66e18 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.711837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c15a9-4f5e-491a-b20d-335965f66e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.712255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c15a9-4f5e-491a-b20d-335965f66e18 admin admin] PATH_INFO: `/v3/projects/8e36d77e2ed94406b072d6cf2b616967/users/97ce9c72842b4293a84392a2e1b66878/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.718237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f3b306-61fc-4db8-b28b-71ddcd48ccf3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:50.724130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db755fa-c602-457b-9e01-85ef053be2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:50.725147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db755fa-c602-457b-9e01-85ef053be2a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.725430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db755fa-c602-457b-9e01-85ef053be2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.725705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db755fa-c602-457b-9e01-85ef053be2a9 admin admin] PATH_INFO: `/v3/projects/61fcbb0ef2724ed29eeb6685dc0eb505/users/7bef754ec9a24b8ea9f64d773771eb1a/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.728868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990c15a9-4f5e-491a-b20d-335965f66e18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e36d77e2ed94406b072d6cf2b616967, user_id=97ce9c72842b4293a84392a2e1b66878, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.731561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990c15a9-4f5e-491a-b20d-335965f66e18 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.741441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db755fa-c602-457b-9e01-85ef053be2a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61fcbb0ef2724ed29eeb6685dc0eb505, user_id=7bef754ec9a24b8ea9f64d773771eb1a, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:50.744128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db755fa-c602-457b-9e01-85ef053be2a9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:50.744768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1021/2058] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:50 2022] PUT /identity/v3/projects/8e36d77e2ed94406b072d6cf2b616967/users/97ce9c72842b4293a84392a2e1b66878/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:50.746109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a430f41-24c0-4788-b8e3-0fceea13cf49 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.746402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a430f41-24c0-4788-b8e3-0fceea13cf49 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.746676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a430f41-24c0-4788-b8e3-0fceea13cf49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.768844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1038/2059] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:50 2022] PUT /identity/v3/projects/61fcbb0ef2724ed29eeb6685dc0eb505/users/7bef754ec9a24b8ea9f64d773771eb1a/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:50.770015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.770313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.770594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.784488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1a430f41-24c0-4788-b8e3-0fceea13cf49 None None] MFA Rules not processed for user `e1356bcc15364d4fa3a8b54ba2105b88`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:50.785735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1a430f41-24c0-4788-b8e3-0fceea13cf49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:50.803632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] MFA Rules not processed for user `97ce9c72842b4293a84392a2e1b66878`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:50.805191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:50.856219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:50.858482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:50.873468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1022/2060] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:50.875381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:50.875751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:50.876099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:50.921789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1039/2061] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:50.922664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:50.923976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:50.933497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] MFA Rules not processed for user `7bef754ec9a24b8ea9f64d773771eb1a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:50.934820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.013793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c58a03b-e352-4409-b4d5-e181ddc8ebe6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.016003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1023/2062] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:51.016881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.018136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.021340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.022060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.022344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.022617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.029849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837db69-d082-438e-81b6-e2447f1bf23a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.033505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.034155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.034426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.034543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.034721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.037301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.044372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.045866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.047003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1040/2063] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:50 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:51.047784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.056349 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1024/2064] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:51 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:51.062854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.115643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf90b-1bfa-4b4c-84ad-378fbfa2e4ff service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.121617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.122634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.123014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.123298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.124068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.126016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.131487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98792779-bc1e-4bff-87db-59c6997504fc service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.137692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.139086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.139373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.139648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] PATH_INFO: `/v3/users/e1356bcc15364d4fa3a8b54ba2105b88` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.140891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1356bcc15364d4fa3a8b54ba2105b88)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.142945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.155517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1041/2065] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:51 2022] POST /identity/v3/projects => generated 387 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:51.159916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.223531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1356bcc15364d4fa3a8b54ba2105b88'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:51.233039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1356bcc15364d4fa3a8b54ba2105b88'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:51.236301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fbc7b1-c2c0-4912-9b6d-8406461d74cd admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.242803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1356bcc15364d4fa3a8b54ba2105b88'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:51.243654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.244547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.245288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.245650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.246890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.250111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.253416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:51.260897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1356bcc15364d4fa3a8b54ba2105b88'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:51.261884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-27e0a198-67f0-4a55-8c9a-8e5b48a0c1d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1356bcc15364d4fa3a8b54ba2105b88'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:51.269070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:51.270063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1025/2066] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:51 2022] DELETE /identity/v3/users/e1356bcc15364d4fa3a8b54ba2105b88 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:51.274543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.274725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.274976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.284176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1042/2067] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:51 2022] POST /identity/v3/users => generated 300 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:51.285428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.288630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.300151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:51.301599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.303106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2fbfa5-558e-4f83-84d8-c7e89b220c45 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.385602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.386294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.386683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.386967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.388167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.390338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.401035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1026/2068] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:51.409241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.411527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1043/2069] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:51 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:51.412588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.415133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.427707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a2a33-05e2-4733-b99c-a508b37e5e67 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.446223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.447153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.447481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.447715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] PATH_INFO: `/v3/projects/e74364728ca34b27988bfb1caee63c46/users/35ca651c800a4547a1bfd72081c89001/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.467710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e74364728ca34b27988bfb1caee63c46, user_id=35ca651c800a4547a1bfd72081c89001, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.471683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.489907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1027/2070] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:51 2022] PUT /identity/v3/projects/e74364728ca34b27988bfb1caee63c46/users/35ca651c800a4547a1bfd72081c89001/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:51.494438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.499009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafb1706-a320-4653-87ee-8371bbea738d admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.551003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.551797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.552149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.552492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.562102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675eb67-0f6d-4832-8a6a-0ed5d3f85cd4 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.562573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.564065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.566715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f5598d-4f91-4e46-95dc-aea613fd7520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.567373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f5598d-4f91-4e46-95dc-aea613fd7520 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.567654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f5598d-4f91-4e46-95dc-aea613fd7520 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.567931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f5598d-4f91-4e46-95dc-aea613fd7520 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.568653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f5598d-4f91-4e46-95dc-aea613fd7520 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.570324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f5598d-4f91-4e46-95dc-aea613fd7520 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.573861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1044/2071] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:51.583401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1028/2072] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:51 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:51.584489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.594434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab608b8c-d37b-4a97-9b4d-5bc8dcea24ee service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.599392 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.600266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.600548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.601421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] PATH_INFO: `/v3/projects/e74364728ca34b27988bfb1caee63c46/users/35ca651c800a4547a1bfd72081c89001/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.603519 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e74364728ca34b27988bfb1caee63c46, user_id=35ca651c800a4547a1bfd72081c89001, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.606232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.617623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1045/2073] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:51 2022] PUT /identity/v3/projects/e74364728ca34b27988bfb1caee63c46/users/35ca651c800a4547a1bfd72081c89001/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:05:51.629662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.629983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.630238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.656921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] MFA Rules not processed for user `35ca651c800a4547a1bfd72081c89001`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:51.658140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.710869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:51.740554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1029/2074] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:51 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:51.753149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.755042 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.795183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:51.815422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e198945a-453b-4278-8927-779f4b25fee9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.819138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecfcfc1-fbc1-4c29-9bd1-fd0dc71f8582 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.820407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecfcfc1-fbc1-4c29-9bd1-fd0dc71f8582 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.820807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecfcfc1-fbc1-4c29-9bd1-fd0dc71f8582 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.821150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecfcfc1-fbc1-4c29-9bd1-fd0dc71f8582 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.830998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecfcfc1-fbc1-4c29-9bd1-fd0dc71f8582 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.832427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecfcfc1-fbc1-4c29-9bd1-fd0dc71f8582 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.842537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1046/2075] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:51 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:51.849762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:51.851736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:51.927088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009fbc1f-e987-465e-8e54-ff0589dd1582 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:51.935415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:51.936296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:51.936581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:51.936913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] PATH_INFO: `/v3/projects/8d0e1fda79e3489685753c227d79c8f2` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:51.938113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d0e1fda79e3489685753c227d79c8f2)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:51.940014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:51.989200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d0e1fda79e3489685753c227d79c8f2', 'request_id': 'req-92369801-fe32-42fc-8c4b-4c0c2b733136'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:51.996440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d0e1fda79e3489685753c227d79c8f2', 'request_id': 'req-92369801-fe32-42fc-8c4b-4c0c2b733136'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:52.008641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d0e1fda79e3489685753c227d79c8f2', 'request_id': 'req-92369801-fe32-42fc-8c4b-4c0c2b733136'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:52.029275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] The token cache is being invalidate because project 8d0e1fda79e3489685753c227d79c8f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:52.029824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:52.030526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-92369801-fe32-42fc-8c4b-4c0c2b733136 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:52.036773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1030/2076] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:51 2022] DELETE /identity/v3/projects/8d0e1fda79e3489685753c227d79c8f2 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:52.037241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.037540 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.037815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.057780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:52.059016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:52.145257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1047/2077] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:52.147092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.147380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.147677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.185316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:52.186598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:52.197488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1031/2078] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:52.202604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.212334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb7f7bc-1d2a-4c16-97a3-c04c367450ed None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.218315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.218995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.219277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.219559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.220385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:52.220752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.222390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.238957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1048/2079] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:05:52.243723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.258431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f70ec-b108-4967-9c04-c6145bc488bb None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.265065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.266022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.266435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.266844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.267741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.270064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.330594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1032/2080] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:52 2022] POST /identity/v3/projects => generated 385 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:52.334887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.345188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316ac600-f3de-41cb-ba79-17cc44fe82b7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.414163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.414920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.415276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.415620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.416366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.418363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.442873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1049/2081] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:52 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:05:52.444045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.454716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddf6829-7ebd-46c3-b045-adf643df7c67 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.462794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.463475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.463748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.464027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.465137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.465683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.468121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.468250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:52.479575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1033/2082] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:52 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:52.483293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.497853 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d9769-96f8-402a-8879-2ef9a667e6c8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.504332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.504833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.505215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.505461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.506156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.508732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.519775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1034/2083] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:52 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:52.523969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.534514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6452d81-fcfd-44f8-91e5-c56811c57b88 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.539910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6114696a-2a25-4548-aced-5eecd02e91de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.540945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114696a-2a25-4548-aced-5eecd02e91de admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.541282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114696a-2a25-4548-aced-5eecd02e91de admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.541562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114696a-2a25-4548-aced-5eecd02e91de admin admin] PATH_INFO: `/v3/projects/a2eec168d39c482284f15c195999f38f/users/746a54e660e143e1b937bb754664bbdb/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.543154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a5f20-725e-47e6-8ccc-bbc9b226e6a1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:52.547043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:52.547765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.548119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.548466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.550554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:52.558002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6114696a-2a25-4548-aced-5eecd02e91de admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2eec168d39c482284f15c195999f38f, user_id=746a54e660e143e1b937bb754664bbdb, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.560787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6114696a-2a25-4548-aced-5eecd02e91de admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.586458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:52.597487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1035/2084] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:52 2022] PUT /identity/v3/projects/a2eec168d39c482284f15c195999f38f/users/746a54e660e143e1b937bb754664bbdb/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:52.619614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:52.619981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:52.620823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:52.645242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:52.686298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:52.688185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] MFA Rules not processed for user `746a54e660e143e1b937bb754664bbdb`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:52.689454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:52.691852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:52.718078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1050/2085] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:52 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:52.801173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1036/2086] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:52 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:52.982000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:52.993198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8af3c-239e-41e5-8fc6-174746fb5421 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:53.113395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5bfee9-1cc5-4b47-a35b-103ae9a2f909 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:53.114057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5bfee9-1cc5-4b47-a35b-103ae9a2f909 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:53.114395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5bfee9-1cc5-4b47-a35b-103ae9a2f909 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:53.114763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5bfee9-1cc5-4b47-a35b-103ae9a2f909 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:53.127041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5bfee9-1cc5-4b47-a35b-103ae9a2f909 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:53.129668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5bfee9-1cc5-4b47-a35b-103ae9a2f909 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:53.141401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1051/2087] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:55.821219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:55.883374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb847178-b405-417a-b410-de3b28c242c3 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:55.887932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:55.888828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:55.889114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:55.889388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] PATH_INFO: `/v3/users/746a54e660e143e1b937bb754664bbdb` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:55.890595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] RBAC: Authorizing `identity:delete_user(user_id=746a54e660e143e1b937bb754664bbdb)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:55.892510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:55.942203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '746a54e660e143e1b937bb754664bbdb'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:55.949792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '746a54e660e143e1b937bb754664bbdb'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:55.957819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '746a54e660e143e1b937bb754664bbdb'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:55.970637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '746a54e660e143e1b937bb754664bbdb'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:55.971345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '746a54e660e143e1b937bb754664bbdb'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:55.977858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1037/2088] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:55 2022] DELETE /identity/v3/users/746a54e660e143e1b937bb754664bbdb => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:55.982383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:55.982882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:55.982995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.000952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:56.002160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.066083 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1052/2089] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:56.076343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:56.077983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.095331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:56.149010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e292527-b84a-43e2-9581-4f180176862f admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:56.152797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc3bd6c-895c-4a27-9b9a-76f8fb2e9d19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:56.153445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc3bd6c-895c-4a27-9b9a-76f8fb2e9d19 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.153731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc3bd6c-895c-4a27-9b9a-76f8fb2e9d19 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.154006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc3bd6c-895c-4a27-9b9a-76f8fb2e9d19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.164081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc3bd6c-895c-4a27-9b9a-76f8fb2e9d19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:56.165548 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc3bd6c-895c-4a27-9b9a-76f8fb2e9d19 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:56.175314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1038/2090] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:56.511455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:56.512951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.526505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607885c2-f3de-481b-90dd-8c831b0f9236 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:56.531963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:56.532877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.533255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.533428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] PATH_INFO: `/v3/projects/a2eec168d39c482284f15c195999f38f` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.535100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2eec168d39c482284f15c195999f38f)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:56.536634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:56.592142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:56.593738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2eec168d39c482284f15c195999f38f', 'request_id': 'req-c0767193-2028-49aa-84f9-e97aece78407'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:56.602410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2eec168d39c482284f15c195999f38f', 'request_id': 'req-c0767193-2028-49aa-84f9-e97aece78407'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:56.611656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2eec168d39c482284f15c195999f38f', 'request_id': 'req-c0767193-2028-49aa-84f9-e97aece78407'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:56.627341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] The token cache is being invalidate because project a2eec168d39c482284f15c195999f38f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:05:56.628114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:56.628821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c0767193-2028-49aa-84f9-e97aece78407 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:56.630406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1053/2091] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:05:56 2022] DELETE /identity/v3/projects/a2eec168d39c482284f15c195999f38f => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:56.637411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.637778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.638325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.657876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:56.659323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.744215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1039/2092] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:56.745922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.746322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.746705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.767927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:56.769456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.779105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1054/2093] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:56.792583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:56.803649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12fc234-f6ad-487a-8e2e-1b64712b7d4d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:56.814313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:56.815007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.815277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.815653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.816294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:05:56.816610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:56.818526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:56.823376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:56.825205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.833593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1040/2094] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:05:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:05:56.837861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:56.848471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300239a-e8db-4d2b-885d-509288a1a068 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:56.854159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:56.854901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.855188 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.855465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.856163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:56.858164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:56.893263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1041/2095] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:05:56 2022] POST /identity/v3/projects => generated 371 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:56.897489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:56.910691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c788a2c-7a17-4507-9b51-4e3d3c34aa74 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:56.976239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a39776-80c1-4443-8ed3-140c7c00d276 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:56.977327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:56.978106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.978400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.978679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.980972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:56.981341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:56.982014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:56.982300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:56.982578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:56.983278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:56.985219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:57.010039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1042/2096] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:05:56 2022] POST /identity/v3/users => generated 286 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:05:57.013886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:57.024429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93484303-9962-46ce-bfa5-0bbba6e28802 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:57.029171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:57.029818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:57.030102 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:57.030380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:57.031161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:57.032837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:57.038108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:57.045633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1043/2097] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:05:57.048448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:57.049235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:57.050693 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:57.059278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76869b7e-1fd3-4d1d-bd80-be64f93a635e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:57.063680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1055/2098] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:05:57.068876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98d944-34d8-41f1-a64e-e07335e89a72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:57.069691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98d944-34d8-41f1-a64e-e07335e89a72 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:57.069982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98d944-34d8-41f1-a64e-e07335e89a72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:57.070260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98d944-34d8-41f1-a64e-e07335e89a72 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:57.071065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98d944-34d8-41f1-a64e-e07335e89a72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:57.072838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98d944-34d8-41f1-a64e-e07335e89a72 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:57.089028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1044/2099] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:05:57 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:05:57.090467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:57.104052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c349583-f85f-4ce6-8333-565742be9271 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:57.110778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:57.111674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:57.111959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:57.112241 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] PATH_INFO: `/v3/projects/375de853371a422f883980296a46ad43/users/3ece5061796245a1b2a6bef1d9d6323a/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:57.141441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] RBAC: Authorizing `identity:create_grant(project_id=375de853371a422f883980296a46ad43, user_id=3ece5061796245a1b2a6bef1d9d6323a, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:57.144119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:57.162832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1056/2100] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:05:57 2022] PUT /identity/v3/projects/375de853371a422f883980296a46ad43/users/3ece5061796245a1b2a6bef1d9d6323a/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:57.164672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:57.165057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:57.165308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:57.184296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:57.185779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:57.259235 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1045/2101] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:05:57.270093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:57.273102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:57.297116 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:05:57.360335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2483eaf-4886-4310-9fb6-1edf64c0c746 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:57.364178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fff62e-6001-4866-b50e-eaa1d0d15027 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:57.364883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fff62e-6001-4866-b50e-eaa1d0d15027 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:57.365170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fff62e-6001-4866-b50e-eaa1d0d15027 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:57.365441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fff62e-6001-4866-b50e-eaa1d0d15027 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:57.376719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fff62e-6001-4866-b50e-eaa1d0d15027 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:57.378265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fff62e-6001-4866-b50e-eaa1d0d15027 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:57.390536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1057/2102] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:05:59.618996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:59.620594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:59.635473 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93432a7-0299-4d7b-805f-8f632fc27021 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:59.640203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:59.641134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:59.641422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:59.641702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] PATH_INFO: `/v3/users/0a3309df0a7848228c82f154224980a0` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:59.642910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a3309df0a7848228c82f154224980a0)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:59.644891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:59.698660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a3309df0a7848228c82f154224980a0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:59.707537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a3309df0a7848228c82f154224980a0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:59.717504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a3309df0a7848228c82f154224980a0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:59.732534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a3309df0a7848228c82f154224980a0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:59.733342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a3309df0a7848228c82f154224980a0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:05:59.740637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1046/2103] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:05:59 2022] DELETE /identity/v3/users/0a3309df0a7848228c82f154224980a0 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:05:59.745038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:59.745857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:59.746388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:59.765444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:05:59.766698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:59.837304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1058/2104] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:05:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:05:59.855483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:05:59.857267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:05:59.934058 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4690507c-85a9-4df0-9335-741da8a22dfa admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:05:59.938433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:05:59.939229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:05:59.939647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:05:59.940024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:05:59.950905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:05:59.952532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:05:59.965293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1047/2105] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:05:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:00.282710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:00.284685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:00.307730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf41648-72a9-4247-8b15-33b1741e4601 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:00.325445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:00.326811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:00.326811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:00.327008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] PATH_INFO: `/v3/projects/04c52dd23a8f414389f5e0171da297cc` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:00.328202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=04c52dd23a8f414389f5e0171da297cc)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:00.330414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:00.389894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04c52dd23a8f414389f5e0171da297cc', 'request_id': 'req-676dfca7-c855-4913-8f80-6e205c64cd1d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.397980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04c52dd23a8f414389f5e0171da297cc', 'request_id': 'req-676dfca7-c855-4913-8f80-6e205c64cd1d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.409378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04c52dd23a8f414389f5e0171da297cc', 'request_id': 'req-676dfca7-c855-4913-8f80-6e205c64cd1d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.428249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] The token cache is being invalidate because project 04c52dd23a8f414389f5e0171da297cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:00.428643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.434493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1059/2106] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:00 2022] DELETE /identity/v3/projects/04c52dd23a8f414389f5e0171da297cc => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:00.438532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:00.447609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:00.466340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2a709-477d-4128-875a-ccc87b1d6d79 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:00.600525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:00.601995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:00.602289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:00.603460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] PATH_INFO: `/v3/users/7bef754ec9a24b8ea9f64d773771eb1a` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:00.609482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bef754ec9a24b8ea9f64d773771eb1a)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:00.613880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:00.697407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bef754ec9a24b8ea9f64d773771eb1a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.709384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bef754ec9a24b8ea9f64d773771eb1a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.733857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bef754ec9a24b8ea9f64d773771eb1a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.751955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bef754ec9a24b8ea9f64d773771eb1a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.752780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bef754ec9a24b8ea9f64d773771eb1a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:00.774589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1048/2107] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:06:00 2022] DELETE /identity/v3/users/7bef754ec9a24b8ea9f64d773771eb1a => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:01.191657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:01.195824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:01.224932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676dfca7-c855-4913-8f80-6e205c64cd1d admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:01.292102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:01.325395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:01.326822 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:01.327119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:01.327402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] PATH_INFO: `/v3/projects/61fcbb0ef2724ed29eeb6685dc0eb505` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:01.328630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=61fcbb0ef2724ed29eeb6685dc0eb505)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:01.330713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:01.417636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61fcbb0ef2724ed29eeb6685dc0eb505', 'request_id': 'req-834d7a99-4179-47d0-9ebf-67b2248acdd0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.425257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61fcbb0ef2724ed29eeb6685dc0eb505', 'request_id': 'req-834d7a99-4179-47d0-9ebf-67b2248acdd0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.435766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61fcbb0ef2724ed29eeb6685dc0eb505', 'request_id': 'req-834d7a99-4179-47d0-9ebf-67b2248acdd0'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.451291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] The token cache is being invalidate because project 61fcbb0ef2724ed29eeb6685dc0eb505 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:01.452098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.453764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.457628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1060/2108] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:01 2022] DELETE /identity/v3/projects/61fcbb0ef2724ed29eeb6685dc0eb505 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:01.462177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:01.463888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:01.484285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265bc6b1-f2bf-40b7-9a09-980d5c784b4e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:01.566066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:01.567222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:01.567778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:01.568157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] PATH_INFO: `/v3/users/35ca651c800a4547a1bfd72081c89001` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:01.569795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35ca651c800a4547a1bfd72081c89001)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:01.572139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:01.608929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:01.636745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35ca651c800a4547a1bfd72081c89001'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.655078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35ca651c800a4547a1bfd72081c89001'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.666122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35ca651c800a4547a1bfd72081c89001'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.681989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35ca651c800a4547a1bfd72081c89001'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.682953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1aa4403-b353-49ce-9d96-7eda239eee80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35ca651c800a4547a1bfd72081c89001'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:01.690370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1049/2109] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:06:01 2022] DELETE /identity/v3/users/35ca651c800a4547a1bfd72081c89001 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:02.052791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.054628 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:02.075345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d7a99-4179-47d0-9ebf-67b2248acdd0 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.192764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.194010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.195134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.195625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] PATH_INFO: `/v3/projects/e74364728ca34b27988bfb1caee63c46` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.197450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] RBAC: Authorizing `identity:delete_project(project_id=e74364728ca34b27988bfb1caee63c46)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.201416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:02.333147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e74364728ca34b27988bfb1caee63c46', 'request_id': 'req-0ffa8103-8a78-4761-a2b3-7727add20bae'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:02.348248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e74364728ca34b27988bfb1caee63c46', 'request_id': 'req-0ffa8103-8a78-4761-a2b3-7727add20bae'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:02.361600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e74364728ca34b27988bfb1caee63c46', 'request_id': 'req-0ffa8103-8a78-4761-a2b3-7727add20bae'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:02.380819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] The token cache is being invalidate because project e74364728ca34b27988bfb1caee63c46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:02.381138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:02.382546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-0ffa8103-8a78-4761-a2b3-7727add20bae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:02.384965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1061/2110] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:02 2022] DELETE /identity/v3/projects/e74364728ca34b27988bfb1caee63c46 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:02.391400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.392236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.392410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.415400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:02.417084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:02.490910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1050/2111] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:02.500332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.500905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.502329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:02.527899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:02.530962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:02.541011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1062/2112] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:02.546605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.557734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad36e9b-61dc-4e5a-ad2b-4c24bf68eed8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.570509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.571553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.572042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.572563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.573538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:06:02.574033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.577923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:02.608130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1051/2113] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:06:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:06:02.612622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.627522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cbb8be-1b45-4fc7-990d-e1ef832dc2ad None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.637269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.637947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.638232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.638520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.640270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.642528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:02.679759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1063/2114] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:06:02 2022] POST /identity/v3/projects => generated 385 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:02.681080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.695911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60f1ca-7c3c-4a66-8476-71a988e6c11b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.774505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.775197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.775490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.775941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.776664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.778730 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:02.814246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1052/2115] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:06:02 2022] POST /identity/v3/users => generated 293 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:02.815210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.829803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80ffc7-d5cd-4512-ac9a-64de90399a85 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.842147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.842834 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.843122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.843403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.844151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.845818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:02.859348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1064/2116] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:02 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:06:02.863497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.878671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2e7fdf-ce56-4a54-b995-8abbdc29d42e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.885975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ee1fdc-6e3d-40e4-b353-a13a56cfc5a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.887057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ee1fdc-6e3d-40e4-b353-a13a56cfc5a8 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.888632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ee1fdc-6e3d-40e4-b353-a13a56cfc5a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.889274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ee1fdc-6e3d-40e4-b353-a13a56cfc5a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.890275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ee1fdc-6e3d-40e4-b353-a13a56cfc5a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.892635 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ee1fdc-6e3d-40e4-b353-a13a56cfc5a8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:02.911957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:02.912136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1053/2117] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:02 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:02.930076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018cd16-f1e9-4a8e-b5e5-35c34c73e147 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:02.945183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:02.946288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:02.946623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:02.947008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] PATH_INFO: `/v3/projects/8bfd655731dd47f5b12021e1e8bb4c54/users/dc81f4b27b1a404b866942450ac4e28d/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:02.971384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bfd655731dd47f5b12021e1e8bb4c54, user_id=dc81f4b27b1a404b866942450ac4e28d, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:02.974617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.004154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1065/2118] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:02 2022] PUT /identity/v3/projects/8bfd655731dd47f5b12021e1e8bb4c54/users/dc81f4b27b1a404b866942450ac4e28d/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:03.019633 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31edf015-6c43-4301-85bd-f8dee4e7042c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.019925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31edf015-6c43-4301-85bd-f8dee4e7042c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.020383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31edf015-6c43-4301-85bd-f8dee4e7042c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.071631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:03.088554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-31edf015-6c43-4301-85bd-f8dee4e7042c None None] MFA Rules not processed for user `dc81f4b27b1a404b866942450ac4e28d`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:03.090846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-31edf015-6c43-4301-85bd-f8dee4e7042c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:03.232671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.235051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1054/2119] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:03.249270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.249628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.249972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.291198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] MFA Rules not processed for user `3ece5061796245a1b2a6bef1d9d6323a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:03.292586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:03.316587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816ef68f-e03d-4351-9f49-872b54ef5573 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.322245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.322993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.323336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.323622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.324318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.326276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.365242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1066/2120] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:06:03 2022] POST /identity/v3/projects => generated 383 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:06:03.369830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.432336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1055/2121] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:03 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:03.437834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.439391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:03.469357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b64359d-d6d3-4d3d-95bf-1f74eba9ce74 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.539380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.540221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.540611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.540938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5b4013-94dd-4195-81cc-e5f810cd40eb admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.541131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.542370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.543982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.547357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.548045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.548658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.555097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.556620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.559119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.583455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1056/2122] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:06:03 2022] POST /identity/v3/projects => generated 373 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:06:03.587898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.590612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1067/2123] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:06:03 2022] POST /identity/v3/users => generated 298 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:06:03.594980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.605672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:03.631536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e39071-6e94-4438-bc4f-e621c611329c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.669829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488614bc-0eb6-4957-9596-db518c861663 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.673308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:03.691173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.691866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.692150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.692426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.693432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.695998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.784134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.785104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.785393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.785678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.786427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.788176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.795047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1057/2124] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:06:03 2022] POST /identity/v3/users => generated 293 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:03.800661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.815953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8224fa-6a7d-4c90-91c3-0abf566b0d90 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.819077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1068/2125] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:03 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:03.824055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.826930 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.827952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.828339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.830044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.831865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.834360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.840738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6b641-bc13-4f36-8972-026994fc4c9e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.854496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.854807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.855121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] PATH_INFO: `/v3/projects/9b75ae677ff84d9bac70a75389fa43fb/users/74eff1c5119b4721a71fa29da4c88e49/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1058/2126] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:03 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:03.868072 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.879516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b75ae677ff84d9bac70a75389fa43fb, user_id=74eff1c5119b4721a71fa29da4c88e49, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.880489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1998f21e-a92c-4a41-99b5-7dc73cd502bb admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:03.882299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.907033 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:03.908053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:03.908341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:03.908618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] PATH_INFO: `/v3/projects/37d05fa2cf3643838d6c6a3139bf3953/users/1640c380a8314c26a0db18e69b206a39/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:03.918310 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1069/2127] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:03 2022] PUT /identity/v3/projects/9b75ae677ff84d9bac70a75389fa43fb/users/74eff1c5119b4721a71fa29da4c88e49/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:03.919436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:03.949085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=37d05fa2cf3643838d6c6a3139bf3953, user_id=1640c380a8314c26a0db18e69b206a39, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:03.952220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:03.985183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1059/2128] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:03 2022] PUT /identity/v3/projects/37d05fa2cf3643838d6c6a3139bf3953/users/1640c380a8314c26a0db18e69b206a39/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:03.989022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:04.055490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e7024-d46a-4681-95e9-02ce77d70949 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:04.079642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a6272-bc1c-401e-9abc-da92def599ea admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:04.089073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:04.089784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:04.090074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:04.090358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:04.091112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:04.092774 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:04.108245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1060/2129] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:03 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:04.118317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:04.135908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:04.136588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f48817a-80a5-4874-bdd8-78122e2a100a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:04.137129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:04.137620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:04.138091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:04.139530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:04.142395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:04.142809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f03b25-323e-4972-9ba6-e9c54ddcb6e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:04.143996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f03b25-323e-4972-9ba6-e9c54ddcb6e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:04.144288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f03b25-323e-4972-9ba6-e9c54ddcb6e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:04.144826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f03b25-323e-4972-9ba6-e9c54ddcb6e0 admin admin] PATH_INFO: `/v3/projects/37d05fa2cf3643838d6c6a3139bf3953/users/1640c380a8314c26a0db18e69b206a39/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:04.148356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f03b25-323e-4972-9ba6-e9c54ddcb6e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37d05fa2cf3643838d6c6a3139bf3953, user_id=1640c380a8314c26a0db18e69b206a39, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:04.152026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f03b25-323e-4972-9ba6-e9c54ddcb6e0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:04.163770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1070/2130] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:03 2022] GET /identity/v3/roles => generated 1420 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:04.167195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:04.168195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1061/2131] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:04 2022] PUT /identity/v3/projects/37d05fa2cf3643838d6c6a3139bf3953/users/1640c380a8314c26a0db18e69b206a39/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:04.184652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821d8b3b-5d40-4151-8b93-2ed74c885714 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:04.222968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:04.270451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:04.272092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:04.273570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:04.275818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] PATH_INFO: `/v3/projects/9b75ae677ff84d9bac70a75389fa43fb/users/74eff1c5119b4721a71fa29da4c88e49/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:04.286502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b75ae677ff84d9bac70a75389fa43fb, user_id=74eff1c5119b4721a71fa29da4c88e49, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:04.290987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:04.311828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1071/2132] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:04 2022] PUT /identity/v3/projects/9b75ae677ff84d9bac70a75389fa43fb/users/74eff1c5119b4721a71fa29da4c88e49/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:04.313935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f057454c-0b6c-4ee5-a122-01692529fe6b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:04.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f057454c-0b6c-4ee5-a122-01692529fe6b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:04.314859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f057454c-0b6c-4ee5-a122-01692529fe6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:04.350881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f057454c-0b6c-4ee5-a122-01692529fe6b None None] MFA Rules not processed for user `74eff1c5119b4721a71fa29da4c88e49`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:04.352387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f057454c-0b6c-4ee5-a122-01692529fe6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:04.461928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:04.534294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1062/2133] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:04 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:04.541271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:04.543459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:04.675052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5851d4ca-c928-4c45-a851-b9167c356478 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:04.680854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:04.682365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:04.682655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:04.682939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:04.778535 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:04.782906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:04.796661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1072/2134] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:04 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:05.251872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b33bbd9-5ffc-41bf-9462-100bf59a7f09 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:05.252237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b33bbd9-5ffc-41bf-9462-100bf59a7f09 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:05.252475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b33bbd9-5ffc-41bf-9462-100bf59a7f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:05.282985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3b33bbd9-5ffc-41bf-9462-100bf59a7f09 None None] MFA Rules not processed for user `e2293054dd2740e781825cbd068d74f7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:05.285832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3b33bbd9-5ffc-41bf-9462-100bf59a7f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:05.431498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1063/2135] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:05 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:10.270033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:10.287498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb92a4-bd38-4f28-bfc7-ded9f41b6fc7 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:10.309108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:10.310030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:10.310490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:10.310939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:10.354453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:10.356220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:10.379522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1073/2136] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:10 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:12.007294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd0e1b6-3551-4608-81ef-6cee95586d5c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:12.007838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd0e1b6-3551-4608-81ef-6cee95586d5c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:12.009121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd0e1b6-3551-4608-81ef-6cee95586d5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:12.100775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6cd0e1b6-3551-4608-81ef-6cee95586d5c None None] MFA Rules not processed for user `1640c380a8314c26a0db18e69b206a39`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:12.102007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6cd0e1b6-3551-4608-81ef-6cee95586d5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:12.285682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1064/2137] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:12 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:12.300190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:12.301748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:12.388611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:12.412328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc723d7f-e61e-476e-997b-c3795f220f8e service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:12.419931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:12.421217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:12.421418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:12.423027 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:12.557750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:12.559502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:12.571915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1074/2138] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:14.553648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:14.554206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:14.554630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:14.589247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] MFA Rules not processed for user `3ece5061796245a1b2a6bef1d9d6323a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:14.590503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:14.609380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1065/2139] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:14.693740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:14.696060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:14.851799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b790278-b1bd-4c9c-90a1-2f3d0a9f2881 service nova] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:14.856598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:14.857515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:14.857820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:14.858067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] PATH_INFO: `/v3/users/f6cb55e4c41f4e6d94156c597a1baad8` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:14.869342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6cb55e4c41f4e6d94156c597a1baad8)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:14.871417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:15.005412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6cb55e4c41f4e6d94156c597a1baad8'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.027740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6cb55e4c41f4e6d94156c597a1baad8'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.041661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6cb55e4c41f4e6d94156c597a1baad8'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.042550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6cb55e4c41f4e6d94156c597a1baad8'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.044204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-28d08cf5-2c21-497f-9157-2b19d7a63eb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6cb55e4c41f4e6d94156c597a1baad8'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.101464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1075/2140] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:06:14 2022] DELETE /identity/v3/users/f6cb55e4c41f4e6d94156c597a1baad8 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:15.732793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:15.763050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d54a1d-7285-4941-aaf3-b89f21fbaa6f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:15.836679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:15.837675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:15.837960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:15.838234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] PATH_INFO: `/v3/projects/205683784b714ab399311aeca15d0cfa` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:15.840138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=205683784b714ab399311aeca15d0cfa)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:15.842698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:15.844496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:15.896368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '205683784b714ab399311aeca15d0cfa', 'request_id': 'req-db71364b-fe23-4a11-960c-84cff772fdb0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.902230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '205683784b714ab399311aeca15d0cfa', 'request_id': 'req-db71364b-fe23-4a11-960c-84cff772fdb0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.909631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '205683784b714ab399311aeca15d0cfa', 'request_id': 'req-db71364b-fe23-4a11-960c-84cff772fdb0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.930129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] The token cache is being invalidate because project 205683784b714ab399311aeca15d0cfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:15.930423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.931204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db71364b-fe23-4a11-960c-84cff772fdb0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:15.933118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1066/2141] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:15 2022] DELETE /identity/v3/projects/205683784b714ab399311aeca15d0cfa => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:15.938639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:15.940189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:15.940391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:15.960272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:15.961546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:15.976746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:16.063271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1076/2142] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:16.071954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.072246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.072523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.096998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:16.108917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.124833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1067/2143] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:16.128760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.149234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6987d232-dc3e-436e-9858-7f2b2d014e98 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.158555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.159355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.159742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.160196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.161490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:06:16.161903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.164880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.165761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.167360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.191256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1077/2144] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:06:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:06:16.197550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.200471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb683d07-930c-4ebb-b1c2-0ed1aef7ac32 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.208861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.209478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd9fbeb-9318-4647-ab19-dfaf5bcacf11 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.210002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.210282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.210554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] PATH_INFO: `/v3/users/dc81f4b27b1a404b866942450ac4e28d` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.215800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.216758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.217258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.217745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.218335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc81f4b27b1a404b866942450ac4e28d)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.219266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.220296 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.221416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.276936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1078/2145] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:06:16 2022] POST /identity/v3/projects => generated 375 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:16.287283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.355207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc81f4b27b1a404b866942450ac4e28d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:16.363010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc81f4b27b1a404b866942450ac4e28d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:16.371550 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc81f4b27b1a404b866942450ac4e28d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:16.381797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d0cd8f-8568-4806-8905-014e0869f1ac admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.387184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.387509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc81f4b27b1a404b866942450ac4e28d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:16.388212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.388318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7a8582c2-8932-42d0-aea4-b6566fb173b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc81f4b27b1a404b866942450ac4e28d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:16.388668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.389066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.389867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.392115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.394734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1068/2146] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:06:16 2022] DELETE /identity/v3/users/dc81f4b27b1a404b866942450ac4e28d => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:16.399273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.399630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.419605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1079/2147] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:06:16 2022] POST /identity/v3/users => generated 288 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:16.423924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.426024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.429758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:16.431199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.448929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e67554-10c2-4595-94da-88134a70337f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.515004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:16.524473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.525877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.526430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.526767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.527657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.529493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.531326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1069/2148] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:16.547751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.547751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.558234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1080/2149] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:06:16.568883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.579976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17632052-97c2-481f-916e-9c0427137e78 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.586658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.587551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.588046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.588362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.594112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.594112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.601215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:16.613842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1081/2150] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:16.615309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.630391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c870b4-61da-44f2-854a-10dafe59f27f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.643264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.644168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.644453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.644581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2932a5-df99-45db-a478-e5a7ff5430da None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.644749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] PATH_INFO: `/v3/projects/54b5133f102c4606ac7981542949b178/users/c0b6f4cfede44b2aa8f4a8dfc1e211d6/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.653107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d13489-5d0a-4ad6-9240-f60de50d72b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.653771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d13489-5d0a-4ad6-9240-f60de50d72b1 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.654051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d13489-5d0a-4ad6-9240-f60de50d72b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.654323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d13489-5d0a-4ad6-9240-f60de50d72b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.663273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] RBAC: Authorizing `identity:create_grant(project_id=54b5133f102c4606ac7981542949b178, user_id=c0b6f4cfede44b2aa8f4a8dfc1e211d6, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.667230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.675899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d13489-5d0a-4ad6-9240-f60de50d72b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.677967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d13489-5d0a-4ad6-9240-f60de50d72b1 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.690563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1070/2151] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:16.702334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1082/2152] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:16 2022] PUT /identity/v3/projects/54b5133f102c4606ac7981542949b178/users/c0b6f4cfede44b2aa8f4a8dfc1e211d6/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:16.706602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.706916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.707174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.748149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:16.750168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.831306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1071/2153] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:16.843620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:16.845625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:16.939164 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfaeb34-2a48-40b0-8c02-2678d5b4234a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:16.942945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:16.943598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:16.943882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:16.944194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:16.953755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:16.955184 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:16.964972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1083/2154] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:17.112631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:17.114463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:17.162341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f910292-e197-4fc6-9f9a-b5d45b65aea2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:17.168383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:17.169317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:17.169698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:17.170065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] PATH_INFO: `/v3/projects/8bfd655731dd47f5b12021e1e8bb4c54` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:17.171376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bfd655731dd47f5b12021e1e8bb4c54)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:17.173554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:17.263721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bfd655731dd47f5b12021e1e8bb4c54', 'request_id': 'req-6b0319a5-2003-49bb-b659-4aabad966277'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.270965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bfd655731dd47f5b12021e1e8bb4c54', 'request_id': 'req-6b0319a5-2003-49bb-b659-4aabad966277'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.282131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bfd655731dd47f5b12021e1e8bb4c54', 'request_id': 'req-6b0319a5-2003-49bb-b659-4aabad966277'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.309777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] The token cache is being invalidate because project 8bfd655731dd47f5b12021e1e8bb4c54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:17.310122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.310845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.313262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1072/2155] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:17 2022] DELETE /identity/v3/projects/8bfd655731dd47f5b12021e1e8bb4c54 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:17.317051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:17.328899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6aaa39-f83a-4acb-9b30-61a836be0d3b service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:17.441740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:17.443676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:17.443676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:17.443676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] PATH_INFO: `/v3/users/74eff1c5119b4721a71fa29da4c88e49` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:17.445537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74eff1c5119b4721a71fa29da4c88e49)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:17.450416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:17.536317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74eff1c5119b4721a71fa29da4c88e49'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.556461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74eff1c5119b4721a71fa29da4c88e49'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.570648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74eff1c5119b4721a71fa29da4c88e49'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.571602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74eff1c5119b4721a71fa29da4c88e49'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.572297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ac530e90-fb28-4c9d-8b9c-a1ef7f42e328 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74eff1c5119b4721a71fa29da4c88e49'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:17.597527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:17.606739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1084/2156] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:06:17 2022] DELETE /identity/v3/users/74eff1c5119b4721a71fa29da4c88e49 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:06:17.965558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:17.967122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:17.982477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0319a5-2003-49bb-b659-4aabad966277 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.011640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:18.070284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.071402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.071873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.072386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] PATH_INFO: `/v3/projects/9b75ae677ff84d9bac70a75389fa43fb` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.074120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b75ae677ff84d9bac70a75389fa43fb)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.076275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.153233 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b75ae677ff84d9bac70a75389fa43fb', 'request_id': 'req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:18.163242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b75ae677ff84d9bac70a75389fa43fb', 'request_id': 'req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:18.178034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b75ae677ff84d9bac70a75389fa43fb', 'request_id': 'req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:18.200981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] The token cache is being invalidate because project 9b75ae677ff84d9bac70a75389fa43fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:18.201284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:18.203175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-db5ceefd-b6b0-4a90-bec8-90f5c0f78b3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:18.206979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1073/2157] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:17 2022] DELETE /identity/v3/projects/9b75ae677ff84d9bac70a75389fa43fb => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:18.212804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.213720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.214213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.249868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:18.252302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:18.335588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1085/2158] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:18.337552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.337869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.338171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.361706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:18.363082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:18.377991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1074/2159] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:18.382667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:18.396175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e509f-dc0f-47be-9faa-811a23721f43 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.402250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.403003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.403389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.403762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.404627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:06:18.405167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.406977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.422449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1086/2160] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:06:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:06:18.432769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:18.438230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b672d-6369-4abd-8d9a-6b829bab233f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.445115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.445473 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.445832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.446553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.448513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.488499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1075/2161] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:06:18 2022] POST /identity/v3/projects => generated 375 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:18.492968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:18.504881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be4bf25-c09a-474b-9c90-ec6da29760f5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.640019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.640720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.641163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.641446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.642303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.644502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.708038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:18.717513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1087/2162] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:06:18 2022] POST /identity/v3/users => generated 288 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:18.737495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce577cc-5140-4aac-9f02-3df31d3bbf1e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.744929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.745814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.746187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.747160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.748529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.751920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.770615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1076/2163] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:18 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:18.775319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:18.789449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4574bd-e733-4d53-b042-d4b31e415eee admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.797183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f70f97-c7b2-40c9-baf8-2c4678dd5109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.798166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f70f97-c7b2-40c9-baf8-2c4678dd5109 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.798742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f70f97-c7b2-40c9-baf8-2c4678dd5109 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.799316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f70f97-c7b2-40c9-baf8-2c4678dd5109 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.800373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f70f97-c7b2-40c9-baf8-2c4678dd5109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.806546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f70f97-c7b2-40c9-baf8-2c4678dd5109 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.824385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1088/2164] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:18 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:06:18.829165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:18.847807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bdc50-d162-41ef-bb2b-16c59298f479 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:18.855106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81043e85-6a37-42f5-b2bc-a5db784d5d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:18.856115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81043e85-6a37-42f5-b2bc-a5db784d5d57 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.856397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81043e85-6a37-42f5-b2bc-a5db784d5d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.856672 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81043e85-6a37-42f5-b2bc-a5db784d5d57 admin admin] PATH_INFO: `/v3/projects/0b5a0cca786340b0879a41961da94e32/users/4e1c85018f6e47619476e9e01213c062/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.877980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81043e85-6a37-42f5-b2bc-a5db784d5d57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b5a0cca786340b0879a41961da94e32, user_id=4e1c85018f6e47619476e9e01213c062, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:18.878600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81043e85-6a37-42f5-b2bc-a5db784d5d57 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:18.903806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1077/2165] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:18 2022] PUT /identity/v3/projects/0b5a0cca786340b0879a41961da94e32/users/4e1c85018f6e47619476e9e01213c062/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:18.908756 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:18.909129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:18.909483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:18.966527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] MFA Rules not processed for user `4e1c85018f6e47619476e9e01213c062`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:18.969010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:19.045925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:19.058335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1089/2166] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:22.175615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28427dfb-2946-4d6b-becc-12d46f28ac33 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:22.176082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28427dfb-2946-4d6b-becc-12d46f28ac33 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:22.176172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28427dfb-2946-4d6b-becc-12d46f28ac33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:22.218150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-28427dfb-2946-4d6b-becc-12d46f28ac33 None None] MFA Rules not processed for user `c0b6f4cfede44b2aa8f4a8dfc1e211d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:22.219387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-28427dfb-2946-4d6b-becc-12d46f28ac33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:22.270255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:22.309656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1078/2167] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:22 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:22.328680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:22.330477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:22.411301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9600f6-28d3-41f7-93e5-8e114454f866 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:22.418099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:22.418760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:22.419043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:22.419319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:22.427819 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:22.429484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:22.438877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1090/2168] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:22 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:06:24.152769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff7dc9-6cd1-40bf-8af0-cb8bc94071be None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:24.153307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff7dc9-6cd1-40bf-8af0-cb8bc94071be None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:24.153431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff7dc9-6cd1-40bf-8af0-cb8bc94071be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:24.171875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-53ff7dc9-6cd1-40bf-8af0-cb8bc94071be None None] MFA Rules not processed for user `c0b6f4cfede44b2aa8f4a8dfc1e211d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:24.173397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-53ff7dc9-6cd1-40bf-8af0-cb8bc94071be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:24.181013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1079/2169] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:24 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:26.200651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:26.202117 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:26.284043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ce93ac-c795-4405-9a89-01fe54b38b31 service nova] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:26.287692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:26.288328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:26.288605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:26.288904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:26.297720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:26.299152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:26.308795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1091/2170] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:26 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:34.165592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=13 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:36.304607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=72 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:40.288423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:40.289028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:40.289243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:40.307307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:40.308507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:40.367668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1080/2171] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:06:40 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:40.414551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:40.416260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:40.484975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f010e-d554-410a-8b06-1fa193f36963 service glance] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:40.490420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d29410-f85d-4ce5-8b0b-eecff81fc0d1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:40.491102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d29410-f85d-4ce5-8b0b-eecff81fc0d1 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:40.491369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d29410-f85d-4ce5-8b0b-eecff81fc0d1 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:40.491643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d29410-f85d-4ce5-8b0b-eecff81fc0d1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:40.501675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d29410-f85d-4ce5-8b0b-eecff81fc0d1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:40.503330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d29410-f85d-4ce5-8b0b-eecff81fc0d1 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:40.515634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1092/2172] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:06:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:06:46.153649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:46.155788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:46.257304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b01fc7-5cb2-43e8-a5c4-60638cc5ac88 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:46.267171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:46.268500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:46.269025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:46.269201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] PATH_INFO: `/v3/users/97ce9c72842b4293a84392a2e1b66878` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:46.275533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97ce9c72842b4293a84392a2e1b66878)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:46.278317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:46.366305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97ce9c72842b4293a84392a2e1b66878'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:46.381489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97ce9c72842b4293a84392a2e1b66878'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:46.394011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97ce9c72842b4293a84392a2e1b66878'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:46.439420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97ce9c72842b4293a84392a2e1b66878'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:46.440864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97ce9c72842b4293a84392a2e1b66878'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:46.452612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1081/2173] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:06:46 2022] DELETE /identity/v3/users/97ce9c72842b4293a84392a2e1b66878 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:46.461265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:46.463609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:46.464240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:46.501163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:46.503918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:46.645676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1093/2174] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:46.652370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:46.653890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:46.695662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:46.779543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8731fd7a-232a-4eb5-b6ad-cb1e97256f62 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:46.793534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa62e59-6882-460a-85bb-4273ff51a9db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:46.798202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa62e59-6882-460a-85bb-4273ff51a9db service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:46.804917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa62e59-6882-460a-85bb-4273ff51a9db service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:46.806757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa62e59-6882-460a-85bb-4273ff51a9db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:46.832831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa62e59-6882-460a-85bb-4273ff51a9db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:46.834599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa62e59-6882-460a-85bb-4273ff51a9db service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:46.866608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1082/2175] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:47.295447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:47.297620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:47.332407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13de363d-15e2-4b99-94e7-0ce13e3b139e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:47.350916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:47.351978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:47.352119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:47.352434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] PATH_INFO: `/v3/projects/8e36d77e2ed94406b072d6cf2b616967` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:47.353725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e36d77e2ed94406b072d6cf2b616967)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:47.355907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:47.421157 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e36d77e2ed94406b072d6cf2b616967', 'request_id': 'req-7e95ae09-df19-4905-92a9-233366fb5020'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:47.434882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e36d77e2ed94406b072d6cf2b616967', 'request_id': 'req-7e95ae09-df19-4905-92a9-233366fb5020'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:47.453486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e36d77e2ed94406b072d6cf2b616967', 'request_id': 'req-7e95ae09-df19-4905-92a9-233366fb5020'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:47.473975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] The token cache is being invalidate because project 8e36d77e2ed94406b072d6cf2b616967 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:06:47.474654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:47.476023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-7e95ae09-df19-4905-92a9-233366fb5020 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:06:47.482097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1094/2176] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:06:47 2022] DELETE /identity/v3/projects/8e36d77e2ed94406b072d6cf2b616967 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:47.487542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:47.488145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:47.488145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:47.514699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:47.516352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:47.600738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1083/2177] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:47.612730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:47.613049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:47.613328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:47.644425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:47.645966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:47.655363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1095/2178] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:47.659646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:47.674628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c44db-dc6d-4054-b3cd-a363313a6c67 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:47.695280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:47.696621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:47.697006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:47.697362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:47.701364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:06:47.701626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:47.703828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:47.750433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1084/2179] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:06:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:06:47.751591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:47.762733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7ab642-c940-4912-87f6-a3cda790ca5e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:47.770380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:47.770380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:47.770380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:47.770815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:47.771589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:47.773894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:47.778427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:47.839707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1096/2180] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:06:47 2022] POST /identity/v3/projects => generated 387 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:47.841673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:47.867862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6170a922-e6be-4f1a-af34-4952b8d1fb7c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:47.999198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:47.999942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:48.000339 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:48.000629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:48.001726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:48.004826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:48.060030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1085/2181] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:06:47 2022] POST /identity/v3/users => generated 294 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:06:48.061707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:48.081961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7091e3-0ee6-429b-910f-caca05f4e262 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:48.094545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:48.095764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:48.096059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:48.096436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:48.097597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:48.099932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:48.121285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1097/2182] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:48.122907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:48.136407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1dd3e5-d6b8-4434-832b-30b929b61d29 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:48.144334 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1397589-70af-46a7-851b-27760b830f0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:48.147070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1397589-70af-46a7-851b-27760b830f0f admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:48.147597 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1397589-70af-46a7-851b-27760b830f0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:48.148341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1397589-70af-46a7-851b-27760b830f0f admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:48.150760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1397589-70af-46a7-851b-27760b830f0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:48.154358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1397589-70af-46a7-851b-27760b830f0f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:48.183789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1086/2183] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:06:48.185039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:48.212866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b3e4c3-c236-4fd6-aeb7-3b791383ac89 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:48.219896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:48.220874 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:48.223141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:48.223141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] PATH_INFO: `/v3/projects/e814e56722014a868a6986d196f1b234/users/b500d16c17ee438592034f3e01fe3ad2/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:48.253631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e814e56722014a868a6986d196f1b234, user_id=b500d16c17ee438592034f3e01fe3ad2, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:48.283185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:48.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1098/2184] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:06:48 2022] PUT /identity/v3/projects/e814e56722014a868a6986d196f1b234/users/b500d16c17ee438592034f3e01fe3ad2/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:06:48.419196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1733b5a7-f36f-46c5-9d28-c0cb221e5b92 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:48.419196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1733b5a7-f36f-46c5-9d28-c0cb221e5b92 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:48.419196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1733b5a7-f36f-46c5-9d28-c0cb221e5b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:48.482900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1733b5a7-f36f-46c5-9d28-c0cb221e5b92 None None] MFA Rules not processed for user `b500d16c17ee438592034f3e01fe3ad2`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:06:48.484172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1733b5a7-f36f-46c5-9d28-c0cb221e5b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:48.514901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:48.593627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1087/2185] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:06:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:06:48.600892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:06:48.603744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:06:48.702044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1316aa-16fc-4c5c-ae5a-a223414c3a42 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:06:48.708853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:06:48.709801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:06:48.710447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:06:48.710890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:06:48.724603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:06:48.727563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:06:48.755695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1099/2186] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:06:48 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:06:58.573076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=17 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:06:58.736991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=47,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:05.925659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e44a7de-925b-4a17-9a50-d129762bf714 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:05.926231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e44a7de-925b-4a17-9a50-d129762bf714 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:05.926231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e44a7de-925b-4a17-9a50-d129762bf714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:05.945386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5e44a7de-925b-4a17-9a50-d129762bf714 None None] MFA Rules not processed for user `b8f2a78f3a3e4b8e8937e3bc39524690`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:05.947388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5e44a7de-925b-4a17-9a50-d129762bf714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:06.002130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1088/2187] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:06.744996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:06.746984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:06.833274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8efa5-702e-4d9f-b5d2-7ce110073ebc service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:06.839572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:06.845014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:06.845442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:06.845860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] PATH_INFO: `/v3/users/4e1c85018f6e47619476e9e01213c062` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:06.847618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e1c85018f6e47619476e9e01213c062)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:06.849813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:06.928097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e1c85018f6e47619476e9e01213c062'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:06.945538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e1c85018f6e47619476e9e01213c062'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:06.968242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e1c85018f6e47619476e9e01213c062'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:06.969427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e1c85018f6e47619476e9e01213c062'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:06.970167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d9f84896-1f1b-4a1f-876f-d297f48e5d23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e1c85018f6e47619476e9e01213c062'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:07.029369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1100/2188] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:07:06 2022] DELETE /identity/v3/users/4e1c85018f6e47619476e9e01213c062 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:07.034701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:07.035411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:07.035411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:07.053558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:07.054414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:07.055108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:07.066874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:07.068123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:07.097675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:07.099492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:07.135976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1089/2189] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:07.146303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:07.148318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:07.199282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1101/2190] 10.222.0.33 () {58 vars in 967 bytes} [Tue Jul 19 11:07:07 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:07.209160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:07.210875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:07.226650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f81641-4f71-4b7a-a8a7-728ef20bda6f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:07.231782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:07.232429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:07.232912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:07.233392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:07.244119 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:07.244639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:07.246433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:07.259669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1090/2191] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:07:07.327765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60008388-b5e4-4ae2-9ed8-ec51d80cb64f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:07.341888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd330b1-5050-46ed-bbaf-40cb4190277e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:07.345173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd330b1-5050-46ed-bbaf-40cb4190277e service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:07.345483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd330b1-5050-46ed-bbaf-40cb4190277e service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:07.345841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd330b1-5050-46ed-bbaf-40cb4190277e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:07.366400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd330b1-5050-46ed-bbaf-40cb4190277e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:07.368062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd330b1-5050-46ed-bbaf-40cb4190277e service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:07.378261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1102/2192] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 19 11:07:07.944905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:07.946901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:07.961544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92abfae-bbec-4ba6-b7a7-044bd47cdf1d service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:07.966776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:07.967784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:07.968060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:07.968346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] PATH_INFO: `/v3/projects/0b5a0cca786340b0879a41961da94e32` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:07.969529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b5a0cca786340b0879a41961da94e32)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:07.971682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.014995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:08.019967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b5a0cca786340b0879a41961da94e32', 'request_id': 'req-13b6dcd0-5442-48cf-bee2-228515cf5615'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:08.026268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b5a0cca786340b0879a41961da94e32', 'request_id': 'req-13b6dcd0-5442-48cf-bee2-228515cf5615'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:08.034254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b5a0cca786340b0879a41961da94e32', 'request_id': 'req-13b6dcd0-5442-48cf-bee2-228515cf5615'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:08.046761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] The token cache is being invalidate because project 0b5a0cca786340b0879a41961da94e32 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:07:08.047384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:08.047696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-13b6dcd0-5442-48cf-bee2-228515cf5615 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:08.049290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1091/2193] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:07:07 2022] DELETE /identity/v3/projects/0b5a0cca786340b0879a41961da94e32 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:08.054072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.054452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.054622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.072595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:08.073841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:08.191187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1103/2194] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:08.198256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.198553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.198836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.226677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:08.229634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:08.240087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1092/2195] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:08.244338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.257109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6471377b-93d7-4111-acf6-418aadad1ae9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.262243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.263064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.263482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.263771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.264511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:07:08.264862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.266581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.283070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1104/2196] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:07:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:07:08.284469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.296630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f8eade-12f0-431b-bb65-c9a29a714a1c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.302281 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.302947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.303268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.303511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.304242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.306214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.345989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1093/2197] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:07:08 2022] POST /identity/v3/projects => generated 375 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:08.349857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.360463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ae303-1fe8-46d5-994b-63aeafdd89bc admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.422406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.423141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.423426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.424489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.426444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.447114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1105/2198] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:07:08 2022] POST /identity/v3/users => generated 288 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:08.451352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.462124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7171b-9659-4966-8ae6-e84c5e8dc60f admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.466715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.467365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.467652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.467929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.468654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.470315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.482409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1094/2199] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:08.486676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.497338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe26ec7-cf9f-42ba-b446-65168ff62a45 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.502737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb788ea-5dec-40d6-80e8-404c66cf3499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.503427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb788ea-5dec-40d6-80e8-404c66cf3499 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.503766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb788ea-5dec-40d6-80e8-404c66cf3499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.504213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb788ea-5dec-40d6-80e8-404c66cf3499 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.505158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb788ea-5dec-40d6-80e8-404c66cf3499 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.506965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb788ea-5dec-40d6-80e8-404c66cf3499 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.517694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1106/2200] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:08.521737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.532377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a7fd45-ea4d-42e1-9994-4cb8e3ffdfc0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.536985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.537848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.538129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.538403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] PATH_INFO: `/v3/projects/2b4575dc494d40fda826dbcd0600a76a/users/49e762b40bfd49859d5820d3e45adb1b/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.552669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b4575dc494d40fda826dbcd0600a76a, user_id=49e762b40bfd49859d5820d3e45adb1b, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.555342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.567814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1095/2201] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:08 2022] PUT /identity/v3/projects/2b4575dc494d40fda826dbcd0600a76a/users/49e762b40bfd49859d5820d3e45adb1b/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:08.586797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.587094 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.587373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.613334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] MFA Rules not processed for user `49e762b40bfd49859d5820d3e45adb1b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:08.614709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:08.623508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:08.681351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1107/2202] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:08 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:08.691949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:08.693778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:08.754614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f2ef9-6aa2-418c-a599-cfa3529e6d5c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:08.758757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:08.759477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:08.759684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:08.759957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:08.768577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:08.770035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:08.779484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1096/2203] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:08 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:09.544951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:09.547034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:09.630125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e26ae4-6a24-413a-836a-2809135f5d9b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:09.634963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:09.635932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:09.636207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:09.636481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] PATH_INFO: `/v3/users/e2293054dd2740e781825cbd068d74f7` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:09.638105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2293054dd2740e781825cbd068d74f7)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:09.640028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:09.684320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2293054dd2740e781825cbd068d74f7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:09.690919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2293054dd2740e781825cbd068d74f7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:09.698082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2293054dd2740e781825cbd068d74f7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:09.698763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2293054dd2740e781825cbd068d74f7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:09.699373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-00fecf47-5d96-452d-b540-265d65de889b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2293054dd2740e781825cbd068d74f7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:09.717267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1108/2204] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:07:09 2022] DELETE /identity/v3/users/e2293054dd2740e781825cbd068d74f7 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:10.021262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.038592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800b30e7-5b02-4adf-90fe-83b1941511cb service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.096566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.097763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.098045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.098326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] PATH_INFO: `/v3/projects/88dc732812fc4fdd8e9c307f3565ba58` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.099513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=88dc732812fc4fdd8e9c307f3565ba58)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.101591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.113944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:10.129284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a1eb7-8fcc-44ff-aa5f-96a879992ab4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.129607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a1eb7-8fcc-44ff-aa5f-96a879992ab4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.129887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a1eb7-8fcc-44ff-aa5f-96a879992ab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.152276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-468a1eb7-8fcc-44ff-aa5f-96a879992ab4 None None] MFA Rules not processed for user `49e762b40bfd49859d5820d3e45adb1b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:10.153668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-468a1eb7-8fcc-44ff-aa5f-96a879992ab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.153937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88dc732812fc4fdd8e9c307f3565ba58', 'request_id': 'req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.160106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88dc732812fc4fdd8e9c307f3565ba58', 'request_id': 'req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.168092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88dc732812fc4fdd8e9c307f3565ba58', 'request_id': 'req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.178667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:10.179524 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] The token cache is being invalidate because project 88dc732812fc4fdd8e9c307f3565ba58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:07:10.179949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.180718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-bd1a2ed3-ef76-4b5a-b1c5-1d5163c19a0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.182363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1097/2205] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:07:10 2022] DELETE /identity/v3/projects/88dc732812fc4fdd8e9c307f3565ba58 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:10.187463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.187796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.188120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.206614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:10.207831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.213927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1109/2206] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:10.268512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1098/2207] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:10.276727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.277032 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.277311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.295208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:10.296421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.304293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1110/2208] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:10.308655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.318899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1febd2d-8149-4b4f-9a52-e564b15a92a0 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.324067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.324741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.325350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.325648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.326389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:07:10.326701 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.328321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.342366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1099/2209] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:07:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:07:10.347430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.356528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cff1a4d-1e9b-4661-99d4-372c2aa50baf None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.361750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.362565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.362843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.363109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.363799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.365731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.384387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.386327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.397860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1111/2210] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:07:10.399672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.402006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee79303-f410-41b9-8345-3514e5102e75 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.469486 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.470374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.470759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.471228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] PATH_INFO: `/v3/users/b500d16c17ee438592034f3e01fe3ad2` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.472576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b500d16c17ee438592034f3e01fe3ad2)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.473372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ebbf32-a8ca-438c-8878-cd81c627ba0d admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.475504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.480149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.481591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.481591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.481835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.482574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.485456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.518520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1112/2211] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/users => generated 286 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:07:10.522488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.524022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:10.534476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a2fc93-ac63-40b9-9675-64855ca5f070 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.540225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.541222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.541716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.542999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.543244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b500d16c17ee438592034f3e01fe3ad2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.543754 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.545614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.562935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b500d16c17ee438592034f3e01fe3ad2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.563703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1113/2212] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:07:10.568608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.580638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c3792e-a74f-426e-bd4a-474d2b01739f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.586764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b500d16c17ee438592034f3e01fe3ad2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.586936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.587591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.587877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.588154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.588913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.590688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.611139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1114/2213] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:07:10.616626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b500d16c17ee438592034f3e01fe3ad2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.618020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ef49fb0b-8bd7-41c9-a9d6-3a9b7700a121 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b500d16c17ee438592034f3e01fe3ad2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:10.618446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.626257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.626447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1100/2214] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:07:10 2022] DELETE /identity/v3/users/b500d16c17ee438592034f3e01fe3ad2 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:07:10.629746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.630047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.630329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.647612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8af7fe3-e276-49ae-8e7e-b076d8bda177 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:10.669078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:10.670333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.744327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d470d10-b203-4e0f-b7fb-055b54554c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:10.753558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d470d10-b203-4e0f-b7fb-055b54554c19 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.754347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d470d10-b203-4e0f-b7fb-055b54554c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.754833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d470d10-b203-4e0f-b7fb-055b54554c19 admin admin] PATH_INFO: `/v3/projects/5c25ae1045fe4976bcbc673deb9b4f06/users/b681b24b2ad14ce09f64a6597cb441d9/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.777867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1101/2215] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:10.787465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d470d10-b203-4e0f-b7fb-055b54554c19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c25ae1045fe4976bcbc673deb9b4f06, user_id=b681b24b2ad14ce09f64a6597cb441d9, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:10.790826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d470d10-b203-4e0f-b7fb-055b54554c19 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:10.797346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:10.798692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:10.801291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.820539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1115/2216] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:10 2022] PUT /identity/v3/projects/5c25ae1045fe4976bcbc673deb9b4f06/users/b681b24b2ad14ce09f64a6597cb441d9/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:07:10.830630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:10.831020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:10.831398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:10.897381 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] MFA Rules not processed for user `b681b24b2ad14ce09f64a6597cb441d9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:10.902312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:10.944649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f68a2-83ed-4f41-a751-c3f083b5784e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:11.001755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1116/2217] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:11.014511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:11.016068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:11.034146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd1e07a-0719-4419-856e-5b26705a7ff0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:11.034806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd1e07a-0719-4419-856e-5b26705a7ff0 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:11.035093 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd1e07a-0719-4419-856e-5b26705a7ff0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:11.035370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd1e07a-0719-4419-856e-5b26705a7ff0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:11.073046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:11.121456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43516e3-1ce7-4a11-a373-96aff7565ecb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:11.125844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:11.126499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:11.126785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:11.127229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:11.147740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:11.149753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:11.165461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1117/2218] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:11 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:11.201112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:11.202109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd1e07a-0719-4419-856e-5b26705a7ff0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:11.206725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd1e07a-0719-4419-856e-5b26705a7ff0 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:11.215544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a414efaa-860b-495e-80ed-acb1e6865a69 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:11.222165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:11.223130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:11.223552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:11.223901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] PATH_INFO: `/v3/users/b681b24b2ad14ce09f64a6597cb441d9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:11.225747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b681b24b2ad14ce09f64a6597cb441d9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:11.227958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:11.234660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1102/2219] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:11.343092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b681b24b2ad14ce09f64a6597cb441d9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.357197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b681b24b2ad14ce09f64a6597cb441d9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.370438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b681b24b2ad14ce09f64a6597cb441d9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.373463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b681b24b2ad14ce09f64a6597cb441d9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.374413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b681b24b2ad14ce09f64a6597cb441d9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.393092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:11.399741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1118/2220] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:07:11 2022] DELETE /identity/v3/users/b681b24b2ad14ce09f64a6597cb441d9 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:11.404351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:11.404814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:11.405052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:11.435848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:11.438464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:11.583001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1103/2221] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:11.594803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:11.595376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:11.669886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75daa768-46d3-4b09-b15a-69e32b6eda79 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:11.673880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:11.674652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:11.674944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:11.675233 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:11.687895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:11.690376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:11.705831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1119/2222] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:11.819067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:11.821147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:11.835210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f364abf-af90-445f-a16c-e87f383d5b13 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:11.840784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:11.841803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:11.842057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:11.842421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] PATH_INFO: `/v3/projects/e814e56722014a868a6986d196f1b234` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:11.844129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e814e56722014a868a6986d196f1b234)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:11.846309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:11.923562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e814e56722014a868a6986d196f1b234', 'request_id': 'req-d6e29b6e-0971-4c4d-8228-3664c9fc6292'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.938201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e814e56722014a868a6986d196f1b234', 'request_id': 'req-d6e29b6e-0971-4c4d-8228-3664c9fc6292'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.950410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e814e56722014a868a6986d196f1b234', 'request_id': 'req-d6e29b6e-0971-4c4d-8228-3664c9fc6292'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.973496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] The token cache is being invalidate because project e814e56722014a868a6986d196f1b234 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:07:11.973932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.975353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d6e29b6e-0971-4c4d-8228-3664c9fc6292 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:11.977244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1104/2223] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:07:11 2022] DELETE /identity/v3/projects/e814e56722014a868a6986d196f1b234 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:12.146895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.148364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:12.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4f1347-2c0d-460b-8310-f366fd4f3d37 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.251793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.252792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.253080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.253353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] PATH_INFO: `/v3/projects/5c25ae1045fe4976bcbc673deb9b4f06` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.254570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c25ae1045fe4976bcbc673deb9b4f06)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.256492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.323793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c25ae1045fe4976bcbc673deb9b4f06', 'request_id': 'req-ed591f5d-c879-4135-9dc2-91770a3caf8a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:12.330632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c25ae1045fe4976bcbc673deb9b4f06', 'request_id': 'req-ed591f5d-c879-4135-9dc2-91770a3caf8a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:12.338464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c25ae1045fe4976bcbc673deb9b4f06', 'request_id': 'req-ed591f5d-c879-4135-9dc2-91770a3caf8a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:12.353723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] The token cache is being invalidate because project 5c25ae1045fe4976bcbc673deb9b4f06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:07:12.353896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:12.354743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ed591f5d-c879-4135-9dc2-91770a3caf8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:12.356678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1120/2224] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:07:12 2022] DELETE /identity/v3/projects/5c25ae1045fe4976bcbc673deb9b4f06 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:12.362621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.362621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.362621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.383574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:12.384888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:12.423260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:12.480129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1105/2225] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:12.483004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.483474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.483835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.528681 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:12.530262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:12.541632 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1121/2226] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:12.545905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.557948 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407074e4-cfc8-474f-aefc-109d414bab68 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.564763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.566127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.566485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.566690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.567673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:07:12.567980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.569828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.583740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1106/2227] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:07:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:07:12.588472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.599836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547cf90a-9220-4dd4-8468-0afe6443d3af None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.606781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.607335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.607674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.607959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.609036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.611209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.654062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1122/2228] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:07:12 2022] POST /identity/v3/projects => generated 385 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:07:12.658120 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.672851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679b7c1-3a7e-4176-b0b1-7c800d21fb81 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.752752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.754108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.754317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.754764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.755642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.759255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.811502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1107/2229] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:07:12 2022] POST /identity/v3/users => generated 293 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:12.815842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.823765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:12.828779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade422e2-8b71-4e03-ad05-9978ea6a0050 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.838165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.838792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.839080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.839367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.840389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.842509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.873041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1123/2230] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:12 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:07:12.873846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.884979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da48a792-bf01-47f8-ae70-2e4ffcef4418 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.896004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb809e5-9c1a-4e2a-9a57-7ac62cfcf72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.896962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb809e5-9c1a-4e2a-9a57-7ac62cfcf72e admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.897288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb809e5-9c1a-4e2a-9a57-7ac62cfcf72e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.897512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb809e5-9c1a-4e2a-9a57-7ac62cfcf72e admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.898254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb809e5-9c1a-4e2a-9a57-7ac62cfcf72e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.900870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb809e5-9c1a-4e2a-9a57-7ac62cfcf72e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.913561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1108/2231] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:12 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:07:12.923434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:12.938129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e53b3-539b-47ab-adf5-363b3b0d8fa4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:12.949629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:12.951006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:12.951450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:12.952049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] PATH_INFO: `/v3/projects/05630eeaf84a4d1bafd213d419f7a97a/users/9e853b49c2c744e89243ec51e7a1a379/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:12.975793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05630eeaf84a4d1bafd213d419f7a97a, user_id=9e853b49c2c744e89243ec51e7a1a379, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:12.979518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:12.998727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1124/2232] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:12 2022] PUT /identity/v3/projects/05630eeaf84a4d1bafd213d419f7a97a/users/9e853b49c2c744e89243ec51e7a1a379/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:13.004346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbcee63-b544-4928-afbc-056be6eb3128 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:13.004642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbcee63-b544-4928-afbc-056be6eb3128 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:13.005024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbcee63-b544-4928-afbc-056be6eb3128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:13.044568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9fbcee63-b544-4928-afbc-056be6eb3128 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:13.045873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9fbcee63-b544-4928-afbc-056be6eb3128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:13.187562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1109/2233] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:13.201045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:13.203302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:13.351169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d2054f-a986-4b1b-af0b-7fc182dd7421 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:13.374773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:13.375837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:13.376340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:13.376831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:13.405246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:13.406720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:13.422223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1125/2234] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:15.549932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57adf7-0d77-44f7-bc3b-ff584f9bd5bd None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:15.553266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57adf7-0d77-44f7-bc3b-ff584f9bd5bd None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:15.554070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57adf7-0d77-44f7-bc3b-ff584f9bd5bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:15.595484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-8c57adf7-0d77-44f7-bc3b-ff584f9bd5bd None None] MFA Rules not processed for user `c95281b28eae450d8678583a929c3b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:15.602239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8c57adf7-0d77-44f7-bc3b-ff584f9bd5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:15.671643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:15.702255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1110/2235] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:18.507225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:18.508764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:18.519871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b4e8b4-fc2e-425e-8fd8-a9912aa4345f service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:18.523618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:18.524284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:18.524569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:18.524989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:18.526749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:18.591518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:18.593003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:18.603363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1126/2236] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:18 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:21.251064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0254b0a-908a-486c-a8b2-f05309520e2c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:21.252262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0254b0a-908a-486c-a8b2-f05309520e2c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:21.252824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0254b0a-908a-486c-a8b2-f05309520e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:21.277351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b0254b0a-908a-486c-a8b2-f05309520e2c None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:21.278971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b0254b0a-908a-486c-a8b2-f05309520e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:21.435374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1111/2237] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:07:21 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:21.461987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:21.463490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:21.600169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7747b31e-5d04-4f95-b099-d8d1be6c2e4a service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:21.620989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:21.622283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:21.623731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:21.624240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:21.633448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:21.645977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:21.648944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:21.675282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1127/2238] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:22.026841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.027231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.027983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.095794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] MFA Rules not processed for user `9e853b49c2c744e89243ec51e7a1a379`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:22.097800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:22.253690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1112/2239] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:22.254343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:22.274258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89843bff-1678-4eab-af07-2f24e9d5d0f3 service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:22.284395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:22.285101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.285658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.286369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:22.288285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:22.375449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1128/2240] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:07:22 2022] POST /identity/v3/projects => generated 385 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:22.378816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:22.400014 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9e2f63-61c3-4070-b0e5-e22092be74d4 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:22.484017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:22.484399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.484721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.485063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.493568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:22.496083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:22.553705 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1113/2241] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:07:22 2022] POST /identity/v3/users => generated 299 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:07:22.557918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:22.579972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cedfa76-ef9c-4d31-811a-fc03f1295f88 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:22.586492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:22.587912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.588368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.588986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.589896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:22.591804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:22.609636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1129/2242] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:22 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:22.613697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:22.629910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e382a77d-8d04-4d12-960e-07b4d7269dcf admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:22.640849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:22.641769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.642336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] PATH_INFO: `/v3/projects/fc778b6ab5134315899752e000cc5989/users/4162e7aaa48d492b84ad9a44e5b9aad5/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.685222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:22.697004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc778b6ab5134315899752e000cc5989, user_id=4162e7aaa48d492b84ad9a44e5b9aad5, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:22.700572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:22.745845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1114/2243] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:22 2022] PUT /identity/v3/projects/fc778b6ab5134315899752e000cc5989/users/4162e7aaa48d492b84ad9a44e5b9aad5/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:22.750232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:22.766888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eaae90-8968-4758-8ded-4c797e1e17c0 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:22.907869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8087b5-c9c7-45c1-8559-4ba8bc2f5d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:22.908562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8087b5-c9c7-45c1-8559-4ba8bc2f5d1a admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.908878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8087b5-c9c7-45c1-8559-4ba8bc2f5d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.909154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8087b5-c9c7-45c1-8559-4ba8bc2f5d1a admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.909898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8087b5-c9c7-45c1-8559-4ba8bc2f5d1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:22.912043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8087b5-c9c7-45c1-8559-4ba8bc2f5d1a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:22.945792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1130/2244] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:22 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:22.947116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:22.962841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d967ae3-5256-4a4a-88e3-b75c44342509 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:22.968931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:22.971174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:22.971717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:22.972232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] PATH_INFO: `/v3/projects/fc778b6ab5134315899752e000cc5989/users/4162e7aaa48d492b84ad9a44e5b9aad5/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:22.983269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc778b6ab5134315899752e000cc5989, user_id=4162e7aaa48d492b84ad9a44e5b9aad5, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:22.987270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:23.023439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1115/2245] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:22 2022] PUT /identity/v3/projects/fc778b6ab5134315899752e000cc5989/users/4162e7aaa48d492b84ad9a44e5b9aad5/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:32.931391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=45,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:33.013594 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=9,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:34.455930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb86ca7-cba8-4359-aa4b-e655dc3930e2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:34.456397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb86ca7-cba8-4359-aa4b-e655dc3930e2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:34.456477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb86ca7-cba8-4359-aa4b-e655dc3930e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:34.533214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-dfb86ca7-cba8-4359-aa4b-e655dc3930e2 None None] MFA Rules not processed for user `4162e7aaa48d492b84ad9a44e5b9aad5`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:34.536448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-dfb86ca7-cba8-4359-aa4b-e655dc3930e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:34.692740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1131/2246] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:34 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:34.707322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:34.710925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:34.836251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f47f1a-112a-4efd-92ff-4b168a0e618b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:34.840506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:34.841177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:34.841464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:34.841837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:35.046654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:35.049385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:35.064867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1116/2247] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:34 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:35.298753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:35.299043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:35.299331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:35.326951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:35.329136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:35.452195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1132/2248] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:07:35 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:35.465620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:35.550236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e059acb-554c-4c8f-9977-aa1172c3717c service glance] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:35.559954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62d20d-d2f7-44f4-8a9b-2ebc97e9363c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:35.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62d20d-d2f7-44f4-8a9b-2ebc97e9363c service swift] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:35.561167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62d20d-d2f7-44f4-8a9b-2ebc97e9363c service swift] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:35.561443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62d20d-d2f7-44f4-8a9b-2ebc97e9363c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:35.576868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62d20d-d2f7-44f4-8a9b-2ebc97e9363c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:35.578760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62d20d-d2f7-44f4-8a9b-2ebc97e9363c service swift] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:35.594379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1117/2249] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:36.456149 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:36.458603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:36.587554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe7b31d-7e93-4a3f-bc29-094f12bf09de None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:36.597920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:36.598594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:36.598883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:36.599162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:36.618303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:36.639316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:36.640980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:36.652740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1133/2250] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:36 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:42.217139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ec71b-12f9-4bca-b9d8-72cc74d7491b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:42.217729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ec71b-12f9-4bca-b9d8-72cc74d7491b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:42.217847 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ec71b-12f9-4bca-b9d8-72cc74d7491b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:42.241200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-143ec71b-12f9-4bca-b9d8-72cc74d7491b None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:42.243531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-143ec71b-12f9-4bca-b9d8-72cc74d7491b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:42.264395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1118/2251] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:07:42 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:42.281954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:42.301737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72306088-e306-4d79-9135-5e7478ee1f0c service placement] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:42.307817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:42.308610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:42.309134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:42.309662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:42.327551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:42.329453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:42.341466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1134/2252] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:42.982738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d272b0ae-3d15-4f71-980f-d46179d982a3 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:42.983030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d272b0ae-3d15-4f71-980f-d46179d982a3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:42.983309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d272b0ae-3d15-4f71-980f-d46179d982a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:43.007828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d272b0ae-3d15-4f71-980f-d46179d982a3 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:43.009087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d272b0ae-3d15-4f71-980f-d46179d982a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:43.017762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1119/2253] 10.222.0.33 () {58 vars in 967 bytes} [Tue Jul 19 11:07:42 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:43.036066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:43.050923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cebe46d-0879-4886-94c6-ee59c742136d service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:43.057399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:43.060523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:43.060944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:43.061293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:43.067067 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:43.072523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:43.074390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:43.085536 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1135/2254] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:43.354890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae48d9a-d698-4fd3-98af-311f38803f25 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:43.354890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae48d9a-d698-4fd3-98af-311f38803f25 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:43.355452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae48d9a-d698-4fd3-98af-311f38803f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:43.380171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-bae48d9a-d698-4fd3-98af-311f38803f25 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:43.381687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bae48d9a-d698-4fd3-98af-311f38803f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:43.401012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1120/2255] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:07:43 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:43.426402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:43.442279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1b0ff4-9601-481a-a679-91b95131a881 service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:43.453108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:43.453777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:43.454142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:43.454425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:43.465259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:43.467247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:43.477711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1136/2256] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:43.980799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:43.981086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:43.981567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:44.001225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:44.002561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:44.011077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1121/2257] 10.222.0.33 () {58 vars in 967 bytes} [Tue Jul 19 11:07:43 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:44.025327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:44.036796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c33136-e132-4b0f-9236-4d327db4670b service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:44.042258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564578b4-a36f-4a84-ac07-fc4bb4dabd7b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:44.042946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564578b4-a36f-4a84-ac07-fc4bb4dabd7b service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:44.043234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564578b4-a36f-4a84-ac07-fc4bb4dabd7b service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:44.043516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564578b4-a36f-4a84-ac07-fc4bb4dabd7b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:44.052648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564578b4-a36f-4a84-ac07-fc4bb4dabd7b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:44.054362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564578b4-a36f-4a84-ac07-fc4bb4dabd7b service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:44.063334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1137/2258] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:46.359306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:46.360834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:46.392940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:46.453018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ac6806-b00a-42f3-aea5-693d6c5e7630 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:46.458709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:46.459742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:46.460202 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:46.461328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] PATH_INFO: `/v3/users/49e762b40bfd49859d5820d3e45adb1b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:46.462864 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] RBAC: Authorizing `identity:delete_user(user_id=49e762b40bfd49859d5820d3e45adb1b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:46.465621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:46.529379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49e762b40bfd49859d5820d3e45adb1b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:46.537669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49e762b40bfd49859d5820d3e45adb1b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:46.546929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49e762b40bfd49859d5820d3e45adb1b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:46.563189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49e762b40bfd49859d5820d3e45adb1b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:46.564801 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49e762b40bfd49859d5820d3e45adb1b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:46.574183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1122/2259] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:07:46 2022] DELETE /identity/v3/users/49e762b40bfd49859d5820d3e45adb1b => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:46.578421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:46.578923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:46.579018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:46.598807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:46.599945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:46.673362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1138/2260] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:46.684054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:46.685617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:46.785485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f67dcb0-edab-401e-9939-6736c956e65a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:46.803067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720eeda-5771-4b25-9ff5-ba7b73feca94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:46.809630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720eeda-5771-4b25-9ff5-ba7b73feca94 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:46.809845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720eeda-5771-4b25-9ff5-ba7b73feca94 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:46.810116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720eeda-5771-4b25-9ff5-ba7b73feca94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:46.833225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720eeda-5771-4b25-9ff5-ba7b73feca94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:46.837545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720eeda-5771-4b25-9ff5-ba7b73feca94 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:46.848539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1123/2261] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:47.113846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.115605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:47.130285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588b8efa-f9fd-437d-9f73-54d9879ccdd6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.136185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.137173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.137695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.138105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] PATH_INFO: `/v3/projects/2b4575dc494d40fda826dbcd0600a76a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.139562 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b4575dc494d40fda826dbcd0600a76a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.142473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.245780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b4575dc494d40fda826dbcd0600a76a', 'request_id': 'req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:47.258091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b4575dc494d40fda826dbcd0600a76a', 'request_id': 'req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:47.269837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b4575dc494d40fda826dbcd0600a76a', 'request_id': 'req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:47.284100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] The token cache is being invalidate because project 2b4575dc494d40fda826dbcd0600a76a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:07:47.284413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:47.285631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e94d81b9-e80b-4ad2-b6c0-ea5a03b872f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:07:47.287841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1139/2262] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:07:47 2022] DELETE /identity/v3/projects/2b4575dc494d40fda826dbcd0600a76a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:47.294079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.294575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.294920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.325720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:47.327063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:47.408629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:47.457379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1124/2263] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:07:47.530316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.530699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.530987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.550827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:47.552287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:47.566922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1140/2264] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:47.571030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.581755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56534c45-ab39-4c9d-8c5c-05f5924d6755 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.586944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.587803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.588107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.588451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.589257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:07:47.589565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.591449 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88374cfe-bbe0-4049-800b-5fe2c427077b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.604825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1125/2265] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:07:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:07:47.609544 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.621105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d700a7fe-a25c-42ba-82c6-872b28b7a9ec None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.627369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.627533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.627769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.628045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.628621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.629020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.629366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.630271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.632529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.646772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] MFA Rules not processed for user `9e853b49c2c744e89243ec51e7a1a379`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:47.648133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:47.667218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1141/2266] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/projects => generated 379 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:47.671212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.708789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:47.726582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1126/2267] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:47.755985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a2ed9c-ff48-45a9-a6ad-de76164e4276 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.760851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.761532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.761820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.762101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.762800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.764765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.796287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1142/2268] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/users => generated 290 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:47.797569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.810362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93431e92-3048-46fb-a2c6-b0b80a3251f8 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.815728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.816400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.816685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.816981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.817720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.819540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.830169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1127/2269] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:47.834393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.847871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca9ac26-e302-4da0-b358-ed7e585c45b8 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.854041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e53ceb-ad2c-4097-984d-55692a4d44c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.854795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e53ceb-ad2c-4097-984d-55692a4d44c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.855243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e53ceb-ad2c-4097-984d-55692a4d44c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.855423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e53ceb-ad2c-4097-984d-55692a4d44c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.856276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e53ceb-ad2c-4097-984d-55692a4d44c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.858196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e53ceb-ad2c-4097-984d-55692a4d44c1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.873587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1143/2270] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:47.877383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:47.890649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41f25dc-d03f-4de8-9a5b-adb4e659ab94 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:47.897472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b6f8b4-eeb6-4f1c-9fbb-144578263ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:47.899169 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b6f8b4-eeb6-4f1c-9fbb-144578263ba3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.899433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b6f8b4-eeb6-4f1c-9fbb-144578263ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.899930 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b6f8b4-eeb6-4f1c-9fbb-144578263ba3 admin admin] PATH_INFO: `/v3/projects/a7109adf88ac4391aee35c150e96e0b1/users/a217e983effa43f3a4f84221984189c5/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.912250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72554bf-35d5-425e-ac4b-c5a19565527d None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.912638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72554bf-35d5-425e-ac4b-c5a19565527d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.913122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72554bf-35d5-425e-ac4b-c5a19565527d None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.916687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1144/2271] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:07:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:07:47.927350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.927644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.927920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:47.934378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b6f8b4-eeb6-4f1c-9fbb-144578263ba3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7109adf88ac4391aee35c150e96e0b1, user_id=a217e983effa43f3a4f84221984189c5, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:47.937413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b6f8b4-eeb6-4f1c-9fbb-144578263ba3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:47.953039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:47.955509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:47.959292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1128/2272] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:47 2022] PUT /identity/v3/projects/a7109adf88ac4391aee35c150e96e0b1/users/a217e983effa43f3a4f84221984189c5/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:47.974325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51bab09-b504-4569-9365-510beac2c192 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:47.980215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51bab09-b504-4569-9365-510beac2c192 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:47.980666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51bab09-b504-4569-9365-510beac2c192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.003304 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f51bab09-b504-4569-9365-510beac2c192 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:48.005078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f51bab09-b504-4569-9365-510beac2c192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:48.089509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1129/2273] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:48.100091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51bab09-b504-4569-9365-510beac2c192 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:48.102157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f51bab09-b504-4569-9365-510beac2c192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:48.105522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1145/2274] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:07:47 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:48.122599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:48.125454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:48.149323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:48.172163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51bab09-b504-4569-9365-510beac2c192 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:48.177248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621e6bfb-ca53-4008-8e7e-1d28f198eecb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:48.178001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621e6bfb-ca53-4008-8e7e-1d28f198eecb service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.178361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621e6bfb-ca53-4008-8e7e-1d28f198eecb service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.178708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621e6bfb-ca53-4008-8e7e-1d28f198eecb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.189159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621e6bfb-ca53-4008-8e7e-1d28f198eecb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:48.189923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff13fd9-c42c-444c-87ec-4298b7706051 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:48.190763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621e6bfb-ca53-4008-8e7e-1d28f198eecb service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:48.193899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1385d64-dc56-4003-a1a9-6529b3611644 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:48.194647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1385d64-dc56-4003-a1a9-6529b3611644 service glance] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.195010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1385d64-dc56-4003-a1a9-6529b3611644 service glance] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.195359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1385d64-dc56-4003-a1a9-6529b3611644 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.201332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1130/2275] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:48.212304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1385d64-dc56-4003-a1a9-6529b3611644 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:48.213843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1385d64-dc56-4003-a1a9-6529b3611644 service glance] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:48.227462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1146/2276] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:48 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:07:48.315525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184c266-945e-4d2e-841e-fe7786359b90 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.316328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184c266-945e-4d2e-841e-fe7786359b90 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.316851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184c266-945e-4d2e-841e-fe7786359b90 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.332121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1131/2277] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:07:48 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:07:48.358988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e7655f-2357-46d8-989a-3308ce0fe6f1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.359616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e7655f-2357-46d8-989a-3308ce0fe6f1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.360288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e7655f-2357-46d8-989a-3308ce0fe6f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.394495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c7e7655f-2357-46d8-989a-3308ce0fe6f1 None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:48.397035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c7e7655f-2357-46d8-989a-3308ce0fe6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:48.418311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1147/2278] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:07:48 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:48.440195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184c266-945e-4d2e-841e-fe7786359b90 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:48.444363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d184c266-945e-4d2e-841e-fe7786359b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:48.477419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184c266-945e-4d2e-841e-fe7786359b90 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:48.483458 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ec6c3b-2cb9-4e55-86cf-ac609e0121b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:48.484209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ec6c3b-2cb9-4e55-86cf-ac609e0121b2 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.484612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ec6c3b-2cb9-4e55-86cf-ac609e0121b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.484989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ec6c3b-2cb9-4e55-86cf-ac609e0121b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.495711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ec6c3b-2cb9-4e55-86cf-ac609e0121b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:48.497636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ec6c3b-2cb9-4e55-86cf-ac609e0121b2 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:48.509107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1132/2279] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:48 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:48.615837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed92d1-99f5-4f5d-8840-d0d15d8b983e None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.617322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed92d1-99f5-4f5d-8840-d0d15d8b983e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.617608 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed92d1-99f5-4f5d-8840-d0d15d8b983e None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.619044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1148/2280] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:07:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:07:48.625872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f03f33a-937d-4527-bc7f-f8c1bab6f0a8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.626377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f03f33a-937d-4527-bc7f-f8c1bab6f0a8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.626814 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f03f33a-937d-4527-bc7f-f8c1bab6f0a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.651055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7f03f33a-937d-4527-bc7f-f8c1bab6f0a8 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:48.652487 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7f03f33a-937d-4527-bc7f-f8c1bab6f0a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:48.665159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1133/2281] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:07:48 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:48.677505 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed92d1-99f5-4f5d-8840-d0d15d8b983e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:48.700345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed92d1-99f5-4f5d-8840-d0d15d8b983e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:48.703989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:48.704633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:48.706169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:48.706169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:48.721713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:48.723239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:48.744076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1149/2282] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:07:48 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:07:50.582044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea59dbb-a668-444d-ab12-5c6ade81538e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:50.582593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea59dbb-a668-444d-ab12-5c6ade81538e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:50.582593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea59dbb-a668-444d-ab12-5c6ade81538e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:50.610223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4ea59dbb-a668-444d-ab12-5c6ade81538e None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:50.612632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4ea59dbb-a668-444d-ab12-5c6ade81538e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:50.731982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1134/2283] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:07:50 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:50.748648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:50.750864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:50.787035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:50.906657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4debe392-e38f-4aef-ab99-247aa370724e service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:50.912981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:50.914197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:50.914475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:50.914758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:50.945299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:50.946920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:50.955954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1150/2284] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:07:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:07:54.881425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:54.882532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:54.883429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:54.962998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] MFA Rules not processed for user `a217e983effa43f3a4f84221984189c5`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:07:54.962998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:07:55.049921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1135/2285] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:07:54 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:07:55.054967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:55.067261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959af208-2e3c-4947-bc3c-a3a107523181 service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:55.073168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:55.073168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:55.073168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:55.073532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:55.074146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:55.076114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:55.112098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1151/2286] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:07:55 2022] POST /identity/v3/projects => generated 379 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:07:55.117117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:55.138413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8800e5-4614-4e48-831c-108bf681e547 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:55.243768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:07:55.256438 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:55.257365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:55.257660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:55.257942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:55.258723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:55.260607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:55.284742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1136/2287] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:07:55 2022] POST /identity/v3/users => generated 296 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:07:55.288973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:55.301831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aacff2-2505-4fde-acbb-b20968c9e98c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:55.307170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:55.307942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:55.308444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:55.308743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:55.309475 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:55.311347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:55.323272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1152/2288] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:55 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:55.327655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:55.338264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44e03bf-cb04-4457-81cd-bb57e94204b0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:55.343378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:55.344333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:55.344677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:55.345044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] PATH_INFO: `/v3/projects/7734ed48187647389961b70ea9d92128/users/38b720fb42284c2d9892d380d7651999/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:55.361944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7734ed48187647389961b70ea9d92128, user_id=38b720fb42284c2d9892d380d7651999, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:55.364675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:55.379838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1137/2289] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:55 2022] PUT /identity/v3/projects/7734ed48187647389961b70ea9d92128/users/38b720fb42284c2d9892d380d7651999/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:07:55.383966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:55.394443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496d0fa9-7731-4549-944b-f5d6d434b41a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:55.453967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4953a8-d69a-45e3-8cc2-1ea13b262b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:55.454658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4953a8-d69a-45e3-8cc2-1ea13b262b3c admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:55.455030 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4953a8-d69a-45e3-8cc2-1ea13b262b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:55.455389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4953a8-d69a-45e3-8cc2-1ea13b262b3c admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:55.456211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4953a8-d69a-45e3-8cc2-1ea13b262b3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:55.458011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4953a8-d69a-45e3-8cc2-1ea13b262b3c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:55.469341 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1153/2290] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:07:55 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:07:55.473243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:07:55.484688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df186f9e-8fa1-45fe-8aca-ef0b1bd2f64c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:07:55.489454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:07:55.490338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:07:55.490614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:07:55.490894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] PATH_INFO: `/v3/projects/7734ed48187647389961b70ea9d92128/users/38b720fb42284c2d9892d380d7651999/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:07:55.493015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] RBAC: Authorizing `identity:create_grant(project_id=7734ed48187647389961b70ea9d92128, user_id=38b720fb42284c2d9892d380d7651999, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:07:55.495907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:07:55.507425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1138/2291] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:07:55 2022] PUT /identity/v3/projects/7734ed48187647389961b70ea9d92128/users/38b720fb42284c2d9892d380d7651999/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:08:02.679501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be705581-efc8-466f-8ff4-a6018269726f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:02.680053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be705581-efc8-466f-8ff4-a6018269726f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:02.680053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be705581-efc8-466f-8ff4-a6018269726f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:02.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-be705581-efc8-466f-8ff4-a6018269726f None None] MFA Rules not processed for user `38b720fb42284c2d9892d380d7651999`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:08:02.726877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-be705581-efc8-466f-8ff4-a6018269726f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:02.855100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:02.889918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1154/2292] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:08:02 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:08:02.898186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:08:02.900077 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:03.021097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81febf6b-9319-4eae-bc46-565151facade admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:08:03.033163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:08:03.033163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:03.033769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:03.034059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:03.116959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:08:03.118742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:08:03.146175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1139/2293] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:08:02 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:08:05.386288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6bca9-ac1d-4b63-a626-146ecdf6beff None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:05.386857 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6bca9-ac1d-4b63-a626-146ecdf6beff None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:05.387263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6bca9-ac1d-4b63-a626-146ecdf6beff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:05.430438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-66c6bca9-ac1d-4b63-a626-146ecdf6beff None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:08:05.433214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-66c6bca9-ac1d-4b63-a626-146ecdf6beff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:05.510919 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1155/2294] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:08:05 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:08:05.523563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:08:05.577779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:05.603682 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffef32e1-abe5-4e9f-8edf-f2890c517e0a service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:08:05.608955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c25192-11ab-4886-a0ab-c779503e543e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:08:05.609830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c25192-11ab-4886-a0ab-c779503e543e service swift] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:05.610214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c25192-11ab-4886-a0ab-c779503e543e service swift] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:05.610472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c25192-11ab-4886-a0ab-c779503e543e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:05.619872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c25192-11ab-4886-a0ab-c779503e543e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:08:05.621506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c25192-11ab-4886-a0ab-c779503e543e service swift] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:08:05.630809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1140/2295] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:08:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:08:15.502979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=28 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:15.627322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=12 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:22.149453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:22.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:22.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:22.170429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] MFA Rules not processed for user `a217e983effa43f3a4f84221984189c5`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:08:22.172272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:22.183503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1156/2296] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:08:22 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:08:31.014858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:31.015372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:31.015475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:31.033149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:08:31.034367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:31.041560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1141/2297] 10.222.0.33 () {58 vars in 967 bytes} [Tue Jul 19 11:08:31 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:08:31.053490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:08:31.066582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9754aa43-ba37-4b7a-8cb9-b193cc8bb1c1 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:08:31.072218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c99fb0d-ba28-406c-b014-f41963a487df service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:08:31.072892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c99fb0d-ba28-406c-b014-f41963a487df service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:31.073183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c99fb0d-ba28-406c-b014-f41963a487df service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:31.073463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c99fb0d-ba28-406c-b014-f41963a487df service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:31.081724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c99fb0d-ba28-406c-b014-f41963a487df service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:08:31.083303 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c99fb0d-ba28-406c-b014-f41963a487df service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:08:31.092291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1157/2298] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:08:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:08:34.346189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:08:34.399691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c606b4af-ba55-4858-b89c-5787c824f504 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:08:34.403211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:08:34.403858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:34.404154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:34.404412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:34.406215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:34.467676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:08:34.469135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:08:34.478646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1142/2299] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:08:34 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:08:41.089382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=9 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:43.055631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5593f557-0472-431f-8e4c-69f1d4e03b80 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:43.056188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5593f557-0472-431f-8e4c-69f1d4e03b80 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:43.056188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5593f557-0472-431f-8e4c-69f1d4e03b80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:43.073212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5593f557-0472-431f-8e4c-69f1d4e03b80 None None] MFA Rules not processed for user `b8f2a78f3a3e4b8e8937e3bc39524690`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:08:43.074410 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5593f557-0472-431f-8e4c-69f1d4e03b80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:08:43.129294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1158/2300] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:08:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:08:44.474353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=53 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:46.078270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:08:46.086763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c61175-1b76-40db-b676-1ef193f8dda0 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:08:46.090366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:08:46.090991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:08:46.091291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:08:46.091544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:08:46.100282 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:08:46.102230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:08:46.111766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1143/2301] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:08:46 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:08:53.122377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=25 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:08:56.107468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=6 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:09:13.066129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:13.066615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:13.066708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:13.084108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] MFA Rules not processed for user `c95281b28eae450d8678583a929c3b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:09:13.085333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:13.139680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1159/2302] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:09:13 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:09:23.132541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=25 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:09:24.191069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:24.200190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ef080-dadc-4b3d-bd1a-c65d82711e12 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:24.203833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:24.204460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:24.204780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:24.205063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:24.206980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:24.222405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:24.224341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a49f8c0-90fb-418a-9110-6ab65500c7f8 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:24.236545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1144/2303] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:24 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:25.354188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:25.363863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224f720-2044-4848-b567-af2bb1b2222b None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:25.367517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:25.368236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:25.368652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:25.369047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:25.370944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:25.384875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:25.386323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:25.397614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1160/2304] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:25 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:31.774719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:31.775217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:31.775409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:31.793416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] MFA Rules not processed for user `246dc188d9a44b1cb38721e3dd8a863a`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:09:31.794720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:31.850107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1145/2305] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:09:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:09:35.392591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:09:40.496885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:40.498358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:40.512204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725dea0-fa8c-4730-9577-e03fc7fba55e service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:40.517815 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:40.518444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:40.518732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:40.519008 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:40.520740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:40.530748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:40.532328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff04c64-88d9-4a81-bd67-8b692553ed95 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:40.541092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1161/2306] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:40 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:41.843246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=31 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:09:44.691799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:44.704735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced9e76f-608a-4165-b7a0-10ef3d3dae5e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:44.710136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:44.710798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:44.711084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:44.711387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:44.713229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:44.723883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:44.725495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580379d1-dfbe-432e-b902-803ff19d5c4a service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:44.735067 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1146/2307] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:44 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:50.537123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:09:51.528594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994292d0-b960-4ac4-bb57-833af59d6cb6 None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:51.528977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994292d0-b960-4ac4-bb57-833af59d6cb6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:51.529293 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994292d0-b960-4ac4-bb57-833af59d6cb6 None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:51.530709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1162/2308] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:09:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:09:51.536890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:51.537440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:51.537440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:51.555819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] MFA Rules not processed for user `ea109e86f77f4660b6538e0ec4676629`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:09:51.557061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:51.563901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1147/2309] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:09:51 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:09:51.578161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994292d0-b960-4ac4-bb57-833af59d6cb6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:51.589109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994292d0-b960-4ac4-bb57-833af59d6cb6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:51.593331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:51.593975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:51.594262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:51.594538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:51.601887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:51.603304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:51.611302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1163/2310] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:51 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:52.014247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:52.016001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:52.084355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac26083f-9b68-42cf-be96-5ace177459b0 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:52.089244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:52.089877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:52.090157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:52.090430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:52.092183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:52.103065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:52.104621 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:52.114119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1148/2311] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:52 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:55.718454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:55.730074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a683cf-8f59-4f89-a972-4a81761462d2 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:55.733773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:55.734466 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:55.734750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:55.735026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:55.736845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:55.747295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:55.748767 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86a7a92-e717-42d7-b4a7-4fa955261f1b service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:55.757240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1164/2312] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:55 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:09:59.589549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:09:59.603610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6680141-bdb7-427a-aa50-7fe9a4107719 service placement] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:09:59.609993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:09:59.610634 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:09:59.610913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:09:59.611185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:09:59.613009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:09:59.627553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:09:59.629262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:09:59.644213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1149/2313] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:09:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:03.120255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e734966-ef72-4599-853e-e549250287ac None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:03.120775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e734966-ef72-4599-853e-e549250287ac None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:03.120981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e734966-ef72-4599-853e-e549250287ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:03.141087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-0e734966-ef72-4599-853e-e549250287ac None None] MFA Rules not processed for user `b8f2a78f3a3e4b8e8937e3bc39524690`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:03.142306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0e734966-ef72-4599-853e-e549250287ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:03.149217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1165/2314] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:03 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:03.458223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:03.467938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4520f356-3f68-4e9f-8715-a57890f10564 service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:03.471741 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:03.472549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:03.473038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:03.473480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:03.475425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:03.485510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:03.486922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:03.494786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1150/2315] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:03 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:06.801039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e734966-ef72-4599-853e-e549250287ac None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:06.802453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0e734966-ef72-4599-853e-e549250287ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:06.870228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e734966-ef72-4599-853e-e549250287ac None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:06.873833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:06.874737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:06.875041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:06.875678 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:06.877132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:06.894652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:06.896415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:06.909198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1166/2316] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:12.162643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:12.171876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=54 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:12.180979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc178e3-ccae-43c3-9348-35396cdd6fd7 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:12.187552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:12.188312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:12.188724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:12.189246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:12.191435 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:12.205268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:12.207291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcac71d-8b52-4a04-9952-4ce9d8960f34 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:12.219493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1151/2317] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:12 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:16.904138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=41 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:22.214411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:30.309071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:30.318349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fabc41f-bdba-4cba-9316-5f529cfad5b3 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:30.321986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:30.322611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:30.322894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:30.323171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:30.324971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:30.387139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:30.388559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f5549-0e3a-4b74-a1b4-236c8a16da65 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:30.398676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1167/2318] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:33.446573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73674832-b7c5-4e40-85bd-ee7546ce5553 None None] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:33.447121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73674832-b7c5-4e40-85bd-ee7546ce5553 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:33.447121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73674832-b7c5-4e40-85bd-ee7546ce5553 None None] PATH_INFO: `/` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:33.448536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1152/2319] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:10:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:10:33.480849 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:33.481371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:33.481476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:33.514358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73674832-b7c5-4e40-85bd-ee7546ce5553 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:33.516039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-73674832-b7c5-4e40-85bd-ee7546ce5553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:33.517821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] MFA Rules not processed for user `8234f48f37ae4384bf6c5f81c664bf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:33.519057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:33.530584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1168/2320] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:10:33 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:33.532631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73674832-b7c5-4e40-85bd-ee7546ce5553 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:33.538668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:33.539591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:33.539876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:33.540213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] PATH_INFO: `/v3/users/c95281b28eae450d8678583a929c3b8a` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:33.541101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:33.541429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c95281b28eae450d8678583a929c3b8a)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:33.542445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:33.543371 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:33.556152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4cade7-d585-4832-9473-8636b1dba7aa None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:33.561404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:33.562046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:33.562332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:33.562603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:33.576007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:33.578589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:33.593422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c95281b28eae450d8678583a929c3b8a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.593875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1169/2321] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:33 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:10:33.602978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c95281b28eae450d8678583a929c3b8a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.616327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c95281b28eae450d8678583a929c3b8a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.634388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c95281b28eae450d8678583a929c3b8a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.634388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e39cba30-a012-4bbe-972b-9bab93267ea7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c95281b28eae450d8678583a929c3b8a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.640328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1153/2322] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:33 2022] DELETE /identity/v3/users/c95281b28eae450d8678583a929c3b8a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:33.821059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:33.832759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5449605b-4e98-47bc-8d16-7370cc4cbfe2 service placement] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:33.893463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:33.894376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:33.894653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:33.894945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] PATH_INFO: `/v3/projects/05fceba2c70c40a399e71daabe8edef6` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:33.896152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] RBAC: Authorizing `identity:delete_project(project_id=05fceba2c70c40a399e71daabe8edef6)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:33.898242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:33.950842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05fceba2c70c40a399e71daabe8edef6', 'request_id': 'req-5bce0726-c4d8-45fa-89fe-1d96f08f3303'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.957804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05fceba2c70c40a399e71daabe8edef6', 'request_id': 'req-5bce0726-c4d8-45fa-89fe-1d96f08f3303'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.966003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05fceba2c70c40a399e71daabe8edef6', 'request_id': 'req-5bce0726-c4d8-45fa-89fe-1d96f08f3303'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.978335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] The token cache is being invalidate because project 05fceba2c70c40a399e71daabe8edef6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:33.978624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.979279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-5bce0726-c4d8-45fa-89fe-1d96f08f3303 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:33.980846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1170/2323] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:33 2022] DELETE /identity/v3/projects/05fceba2c70c40a399e71daabe8edef6 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:33.985823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:33.986113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:33.986411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.003902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:34.005132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:34.063685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1154/2324] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:34.071740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.072033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.072317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.080615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:34.091948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:34.093703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:34.101745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1171/2325] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:34.105929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.115479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f92fe93-219b-4ec1-95e5-5dee3f0d416e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.119924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.120564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.120886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.121154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.121871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:10:34.122174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.123746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.135293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1155/2326] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:10:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:10:34.139779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.148671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f60ff-c13c-41f7-ac49-6cc26f4a13b9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.153251 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.153863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.154142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.154416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.155673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.157583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.182115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1172/2327] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:34 2022] POST /identity/v3/projects => generated 373 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:34.186459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.196103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2320703a-1792-4818-bfa0-8bc9551bc787 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.251443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.252113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.252398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.252674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.253447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.255407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.262248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:34.281642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1156/2328] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:34 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:34.282794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.292912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a985628-a7cc-4086-a563-c3d0b7c6dda8 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.297516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.298168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.298452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.298726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.299449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.301109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.312254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1173/2329] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:34.316303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbdb08-1989-41b8-a9e4-ec6dcd7892fa admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.334365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810f063a-84b4-4c7b-b17a-f9f5f7e87149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.335032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f063a-84b4-4c7b-b17a-f9f5f7e87149 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.335332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f063a-84b4-4c7b-b17a-f9f5f7e87149 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.335607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f063a-84b4-4c7b-b17a-f9f5f7e87149 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.336337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810f063a-84b4-4c7b-b17a-f9f5f7e87149 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.337956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810f063a-84b4-4c7b-b17a-f9f5f7e87149 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.349046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1157/2330] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:34.353653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.363948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b85f74-c4c9-40c1-9ab4-5ccd121e13b5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.370373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.371322 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.371734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.372006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] PATH_INFO: `/v3/projects/d916fb4d05694bd5ba3f9d61678fba03/users/5f95631bc447497ba062a57c877a133b/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.387702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d916fb4d05694bd5ba3f9d61678fba03, user_id=5f95631bc447497ba062a57c877a133b, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.391021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.403962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1174/2331] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:34 2022] PUT /identity/v3/projects/d916fb4d05694bd5ba3f9d61678fba03/users/5f95631bc447497ba062a57c877a133b/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:34.414613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.414898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.415199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.442743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] MFA Rules not processed for user `5f95631bc447497ba062a57c877a133b`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:34.443934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:34.507132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1158/2332] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:34 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:34.518723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.520483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:34.600904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ec9088-777e-4dd6-b119-be6e5fe89fa7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.606432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088112a1-2cd4-4946-a4d6-0fca571ead96 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.607076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088112a1-2cd4-4946-a4d6-0fca571ead96 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.607358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088112a1-2cd4-4946-a4d6-0fca571ead96 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.607633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088112a1-2cd4-4946-a4d6-0fca571ead96 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.615777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088112a1-2cd4-4946-a4d6-0fca571ead96 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.617391 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088112a1-2cd4-4946-a4d6-0fca571ead96 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.626161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1175/2333] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:34 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:34.837598 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:34.903665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749d4979-7c74-4f5f-9d2e-93d28ad9665b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:34.908838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:34.909702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:34.909980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:34.910248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] PATH_INFO: `/v3/users/5f95631bc447497ba062a57c877a133b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:34.911433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f95631bc447497ba062a57c877a133b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:34.913381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:34.959555 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f95631bc447497ba062a57c877a133b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:34.966541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f95631bc447497ba062a57c877a133b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:34.974184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f95631bc447497ba062a57c877a133b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:34.979224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:34.988897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f95631bc447497ba062a57c877a133b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:34.989680 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f95631bc447497ba062a57c877a133b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:34.995884 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1159/2334] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:34 2022] DELETE /identity/v3/users/5f95631bc447497ba062a57c877a133b => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:35.000554 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.000872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.001150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.018063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:35.019268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:35.077473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1176/2335] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:35.087111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:35.088501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:35.147585 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc42843-b327-455a-88b0-eabaaf9aa56e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:35.151291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df4d1b-8820-457a-b9c4-122d4d8466e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:35.151929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df4d1b-8820-457a-b9c4-122d4d8466e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.152354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df4d1b-8820-457a-b9c4-122d4d8466e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.152710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df4d1b-8820-457a-b9c4-122d4d8466e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.166227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df4d1b-8820-457a-b9c4-122d4d8466e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:35.167876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df4d1b-8820-457a-b9c4-122d4d8466e8 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:35.178094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1160/2336] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:35.489674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:35.491350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:35.504053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa4896-a973-4983-b7db-a31b16270093 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:35.509075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:35.510074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.510424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.510774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] PATH_INFO: `/v3/projects/d916fb4d05694bd5ba3f9d61678fba03` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.511993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d916fb4d05694bd5ba3f9d61678fba03)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:35.514503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:35.541130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:35.575432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd916fb4d05694bd5ba3f9d61678fba03', 'request_id': 'req-58b7ab54-05f2-44a1-aac7-b12e44a517c6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:35.583909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd916fb4d05694bd5ba3f9d61678fba03', 'request_id': 'req-58b7ab54-05f2-44a1-aac7-b12e44a517c6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:35.598697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd916fb4d05694bd5ba3f9d61678fba03', 'request_id': 'req-58b7ab54-05f2-44a1-aac7-b12e44a517c6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:35.610742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] The token cache is being invalidate because project d916fb4d05694bd5ba3f9d61678fba03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:35.611023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:35.611651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-58b7ab54-05f2-44a1-aac7-b12e44a517c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:35.613428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1177/2337] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:35 2022] DELETE /identity/v3/projects/d916fb4d05694bd5ba3f9d61678fba03 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:35.618105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.618604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.618719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.642340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:35.644111 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:35.715576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1161/2338] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:35.724275 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.724573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.724911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.744629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:35.745891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:35.755239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1178/2339] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:35.759591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:35.769608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c92a1-f1df-49e3-b04a-7e36af435036 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:35.774271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:35.774944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.775227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.775499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.776247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:10:35.776551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:35.778199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:35.791447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1162/2340] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:10:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:10:35.796729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:35.806393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5782af-801a-458c-9f81-d61aaa2c792c None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:35.811198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:35.811871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.812159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.812436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.813177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:35.815181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:35.843867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1179/2341] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:35 2022] POST /identity/v3/projects => generated 387 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:35.848079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:35.859258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd8ed2-ae79-4a80-a1ce-a90d7048c8cf admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:35.921733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:35.922409 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.922696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.922979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.923710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:35.925678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:35.947436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1163/2342] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:35 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:35.954122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:35.965459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586d305d-d892-4a23-865f-bb20fec5109b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:35.970243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:35.970839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:35.971123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:35.971396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:35.972121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:35.973751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:35.987790 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1180/2343] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:35.991732 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.002556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd716c-61be-4d3c-8b20-9e7646d321e5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.007263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.007924 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.008212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.008489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.009258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.010855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.012226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:36.025057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1164/2344] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:36.028816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.039114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad529e6-9080-48a8-a58f-fe757d322846 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.044071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b518f50-839c-43ec-bc12-fff367361ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.045590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b518f50-839c-43ec-bc12-fff367361ef2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.045965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b518f50-839c-43ec-bc12-fff367361ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.046302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b518f50-839c-43ec-bc12-fff367361ef2 admin admin] PATH_INFO: `/v3/projects/9e1e8b8f0ad54baaba2035513ad79c0c/users/f20b4b3faea348e28900c00930f22905/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.063781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b518f50-839c-43ec-bc12-fff367361ef2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e1e8b8f0ad54baaba2035513ad79c0c, user_id=f20b4b3faea348e28900c00930f22905, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.066897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b518f50-839c-43ec-bc12-fff367361ef2 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.087734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1181/2345] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:36 2022] PUT /identity/v3/projects/9e1e8b8f0ad54baaba2035513ad79c0c/users/f20b4b3faea348e28900c00930f22905/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:36.093348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.098569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:36.110817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.111335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.111818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.162598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] MFA Rules not processed for user `f20b4b3faea348e28900c00930f22905`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:36.164604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:36.243354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b60fe1-7924-4f06-908b-5b9de1be8ac4 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.303945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1182/2346] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:36.311707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.364170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.365511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.365805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.366086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.368096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:36.393883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.397097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.400550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:DELETE=7,keystone:SELECT=90,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:36.413640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1165/2347] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:10:36.432969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d310c-5a60-4656-867f-c6dbda9b40d9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.438368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.439172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.439629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.439911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.440621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.442928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.477035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1183/2348] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:36 2022] POST /identity/v3/projects => generated 385 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:36.480995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.492223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26265b5-c560-45d7-a1d3-ee51882603a0 service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.561733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.562498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.562857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.563212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.563981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.566030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.602891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.612813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1166/2349] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:36 2022] POST /identity/v3/users => generated 299 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:36.615104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67245775-3afe-4ba0-b9dd-9e739af29e39 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.621473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.622474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.622852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.623753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.624905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.626614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.643041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.648367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1184/2350] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:36.653628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ab2cf-6784-4db6-9b6f-7610f4241864 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.659636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.661074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.661074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.661503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] PATH_INFO: `/v3/projects/bfe9514d41584761a476690e1b421b56/users/f124c22ca4c444548c0024192c44aa2a/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.665223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:36.679443 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe9514d41584761a476690e1b421b56, user_id=f124c22ca4c444548c0024192c44aa2a, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.688878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.706009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.706627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1167/2351] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:36 2022] PUT /identity/v3/projects/bfe9514d41584761a476690e1b421b56/users/f124c22ca4c444548c0024192c44aa2a/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:36.721005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cde09cf-3324-46b4-ac3f-ea965dd3f216 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.788217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51caf2e9-2135-48f4-84cc-b93ecc0019e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.788603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51caf2e9-2135-48f4-84cc-b93ecc0019e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.789033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51caf2e9-2135-48f4-84cc-b93ecc0019e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.789415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51caf2e9-2135-48f4-84cc-b93ecc0019e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.790480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51caf2e9-2135-48f4-84cc-b93ecc0019e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.798569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51caf2e9-2135-48f4-84cc-b93ecc0019e8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.815317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:36.821619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1185/2352] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:36.828097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f8bf8-af3c-4507-b85e-0362511c71a6 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:36.834416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:36.835623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.835929 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.836214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] PATH_INFO: `/v3/projects/bfe9514d41584761a476690e1b421b56/users/f124c22ca4c444548c0024192c44aa2a/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.839084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe9514d41584761a476690e1b421b56, user_id=f124c22ca4c444548c0024192c44aa2a, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:36.842559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:36.856824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1168/2353] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:36 2022] PUT /identity/v3/projects/bfe9514d41584761a476690e1b421b56/users/f124c22ca4c444548c0024192c44aa2a/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:36.867011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:36.867306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:36.867587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:36.918860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] MFA Rules not processed for user `f124c22ca4c444548c0024192c44aa2a`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:36.920256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:37.043494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1186/2354] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:36 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:37.050575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:37.053080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:37.116004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35859820-1369-4b40-9940-05b50454ee50 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:37.119624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5482df-12eb-42f8-96d4-ec452a51bf3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:37.120276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5482df-12eb-42f8-96d4-ec452a51bf3c service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:37.120559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5482df-12eb-42f8-96d4-ec452a51bf3c service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:37.120898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5482df-12eb-42f8-96d4-ec452a51bf3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:37.130572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5482df-12eb-42f8-96d4-ec452a51bf3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:37.131992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5482df-12eb-42f8-96d4-ec452a51bf3c service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:37.142678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1169/2355] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:37 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:37.205905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:37.239266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:37.273637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730ffc0a-9749-4869-9e1c-d1d7564217df None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:37.278707 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:37.280393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:37.280794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:37.281136 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] PATH_INFO: `/v3/users/f20b4b3faea348e28900c00930f22905` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:37.282356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f20b4b3faea348e28900c00930f22905)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:37.284287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:37.345023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f20b4b3faea348e28900c00930f22905'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:37.352443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f20b4b3faea348e28900c00930f22905'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:37.361348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f20b4b3faea348e28900c00930f22905'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:37.362074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f20b4b3faea348e28900c00930f22905'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:37.362692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f20b4b3faea348e28900c00930f22905'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:37.380379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1187/2356] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:37 2022] DELETE /identity/v3/users/f20b4b3faea348e28900c00930f22905 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:37.384968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:37.385356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:37.385537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:37.404032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:37.405762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:37.468983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1170/2357] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:37.479370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:37.480944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:37.567698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdba46b3-bcfa-40b6-90d4-600733a93600 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:37.571727 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:37.572486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:37.572914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:37.573734 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:37.586126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:37.587843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:37.604887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1188/2358] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:37.905767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:37.907843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:37.938360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffab7e4-24a4-4cd8-a1fb-c4a2813a917e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:37.943729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:37.945357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:37.945357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:37.945357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] PATH_INFO: `/v3/projects/9e1e8b8f0ad54baaba2035513ad79c0c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:37.946492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e1e8b8f0ad54baaba2035513ad79c0c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:37.948403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:38.002581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e1e8b8f0ad54baaba2035513ad79c0c', 'request_id': 'req-3fb583f2-db97-4078-88ae-5256b710f662'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.009846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e1e8b8f0ad54baaba2035513ad79c0c', 'request_id': 'req-3fb583f2-db97-4078-88ae-5256b710f662'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.020934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e1e8b8f0ad54baaba2035513ad79c0c', 'request_id': 'req-3fb583f2-db97-4078-88ae-5256b710f662'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.029297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:38.034916 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] The token cache is being invalidate because project 9e1e8b8f0ad54baaba2035513ad79c0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:38.035623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.036808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.041768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1171/2359] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:37 2022] DELETE /identity/v3/projects/9e1e8b8f0ad54baaba2035513ad79c0c => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:38.043928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:38.055097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24cf6eb-bc83-45d4-8d84-86425bf55de7 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:38.114751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:38.129261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:38.130142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.130421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.130694 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] PATH_INFO: `/v3/users/f124c22ca4c444548c0024192c44aa2a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.131885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=f124c22ca4c444548c0024192c44aa2a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:38.133961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:38.180189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f124c22ca4c444548c0024192c44aa2a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.187552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f124c22ca4c444548c0024192c44aa2a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.194515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f124c22ca4c444548c0024192c44aa2a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.195292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f124c22ca4c444548c0024192c44aa2a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.195953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be89d50d-341e-4373-af21-cdcbd85305cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f124c22ca4c444548c0024192c44aa2a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.214321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1189/2360] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:38 2022] DELETE /identity/v3/users/f124c22ca4c444548c0024192c44aa2a => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:38.518531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:38.521495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:38.538783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb583f2-db97-4078-88ae-5256b710f662 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:38.600790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:38.601695 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.602031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.602310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] PATH_INFO: `/v3/projects/bfe9514d41584761a476690e1b421b56` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.603529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfe9514d41584761a476690e1b421b56)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:38.605501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:38.652834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfe9514d41584761a476690e1b421b56', 'request_id': 'req-6f56858e-73f3-4b54-8281-036ee71a47e4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.658896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfe9514d41584761a476690e1b421b56', 'request_id': 'req-6f56858e-73f3-4b54-8281-036ee71a47e4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.666584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfe9514d41584761a476690e1b421b56', 'request_id': 'req-6f56858e-73f3-4b54-8281-036ee71a47e4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.677792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] The token cache is being invalidate because project bfe9514d41584761a476690e1b421b56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:38.678087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.678719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6f56858e-73f3-4b54-8281-036ee71a47e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:38.680251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1172/2361] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:38 2022] DELETE /identity/v3/projects/bfe9514d41584761a476690e1b421b56 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:38.685183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.686140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.686278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.706156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:38.707465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:38.768326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1190/2362] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:38.776165 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.776452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.776761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.795041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:38.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:38.805388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1173/2363] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:38.809613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:38.819744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919bef5a-f704-4be5-b451-c1c34178afeb None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:38.824516 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:38.825283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.825615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.825953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.827773 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:10:38.828077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:38.829684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:38.841326 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1191/2364] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:10:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:10:38.845583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:38.854653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e5cce8-fc9d-4c48-8b9d-9d0b513c81b9 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:38.859383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:38.860021 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.860305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.860577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.861332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:38.863217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:38.889710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1174/2365] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:38 2022] POST /identity/v3/projects => generated 393 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:38.893762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:38.903558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5c431d-4c84-4aa6-a621-ae771cbb36ed admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:38.959690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:38.960359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:38.960645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:38.960955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:38.961642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:38.963567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:38.970651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:38.985250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1192/2366] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:38 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:10:38.989320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:38.999687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91e4850-05cc-4ff8-b57a-f2cfac8a2c0f admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.004220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.004908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.005192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.005465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.006209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.007809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.017954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1175/2367] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:39.021649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.031757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70e86c6-c020-43b2-bd90-32269ca9b7cb admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.036288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088e0247-e5b7-41a4-b3dd-c050ce58189e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.036958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088e0247-e5b7-41a4-b3dd-c050ce58189e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.037238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088e0247-e5b7-41a4-b3dd-c050ce58189e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.037513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088e0247-e5b7-41a4-b3dd-c050ce58189e admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.038226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088e0247-e5b7-41a4-b3dd-c050ce58189e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.039799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088e0247-e5b7-41a4-b3dd-c050ce58189e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.049916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1193/2368] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:10:39.053789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.065641 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496672e1-c650-4a58-a10d-060bc955f416 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.071331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.072344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.072636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.073362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] PATH_INFO: `/v3/projects/b34ccf97aacd4f1193e8fec271d4a082/users/121b6839e5f64256a3342961494cade5/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.090091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b34ccf97aacd4f1193e8fec271d4a082, user_id=121b6839e5f64256a3342961494cade5, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.094207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.108610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1176/2369] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:39 2022] PUT /identity/v3/projects/b34ccf97aacd4f1193e8fec271d4a082/users/121b6839e5f64256a3342961494cade5/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:39.119825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.120122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.120403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.149626 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] MFA Rules not processed for user `121b6839e5f64256a3342961494cade5`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:39.150892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:39.216860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1194/2370] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:39.221008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.252951 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:39.293731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6c05f2-9df8-4724-b04d-8dad83d84b70 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.298895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.299543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.299824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.300098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.300811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.302736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.333567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1177/2371] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:39 2022] POST /identity/v3/projects => generated 391 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:39.338107 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.349816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b5eb76-26ff-4a52-8602-6291019c0795 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.412873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.413538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.413867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.414155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.414908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.416830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.438395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1195/2372] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:39 2022] POST /identity/v3/users => generated 302 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:10:39.442452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.452836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36526642-9879-40ea-b0da-b892c937546c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.459799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.460564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.460949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.461295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.462103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.463770 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.478230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1178/2373] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:39.482586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.495090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799903a7-5238-453b-bb5a-0051b5e89559 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.503236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.505028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.505397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.505866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] PATH_INFO: `/v3/projects/7c9d667f2d5a428cb1c46b7a15b56e0e/users/f5171215deb946108a78ed7c59948633/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.522812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c9d667f2d5a428cb1c46b7a15b56e0e, user_id=f5171215deb946108a78ed7c59948633, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.525644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.539813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1196/2374] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:39 2022] PUT /identity/v3/projects/7c9d667f2d5a428cb1c46b7a15b56e0e/users/f5171215deb946108a78ed7c59948633/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:39.543969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.554652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f35aca3-3630-44cc-9313-d20a40feeb0b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.613195 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd39d56-b377-4a3a-a8eb-c3671f86aa23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.613783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd39d56-b377-4a3a-a8eb-c3671f86aa23 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.614066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd39d56-b377-4a3a-a8eb-c3671f86aa23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.614341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd39d56-b377-4a3a-a8eb-c3671f86aa23 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.615075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd39d56-b377-4a3a-a8eb-c3671f86aa23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.616664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd39d56-b377-4a3a-a8eb-c3671f86aa23 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.626936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1179/2375] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:39 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:10:39.631193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.641424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72794dd9-3520-4e43-926d-97728eb843f3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.646088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.647023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.647315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.647594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] PATH_INFO: `/v3/projects/7c9d667f2d5a428cb1c46b7a15b56e0e/users/f5171215deb946108a78ed7c59948633/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.649612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c9d667f2d5a428cb1c46b7a15b56e0e, user_id=f5171215deb946108a78ed7c59948633, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.652232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.663121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1197/2376] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:39 2022] PUT /identity/v3/projects/7c9d667f2d5a428cb1c46b7a15b56e0e/users/f5171215deb946108a78ed7c59948633/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:39.673838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.674133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.674418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.702644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] MFA Rules not processed for user `f5171215deb946108a78ed7c59948633`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:39.703875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:39.771881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1180/2377] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:39 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:39.782929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.784673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:39.807502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:39.843375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72379202-5e34-4aee-82e4-58d8c6375e81 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.847079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.847721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.848026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.848289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:39.859140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:39.868873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1198/2378] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:39 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:39.914834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:39.923265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:39.949581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22f150d-a7e3-4d66-8ee6-6acd4991846e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:39.955032 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71089203-b96f-4e64-a0c2-29677d40b2ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:39.955785 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71089203-b96f-4e64-a0c2-29677d40b2ae service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:39.955982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71089203-b96f-4e64-a0c2-29677d40b2ae service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:39.956256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71089203-b96f-4e64-a0c2-29677d40b2ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:39.978224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:40.029034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71089203-b96f-4e64-a0c2-29677d40b2ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:40.030708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71089203-b96f-4e64-a0c2-29677d40b2ae service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:40.039846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1181/2379] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:43.488281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:43.581162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eba8d9a-a356-4870-82a9-39d8c50750b3 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:43.587480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:43.588357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:43.588640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:43.588952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] PATH_INFO: `/v3/users/121b6839e5f64256a3342961494cade5` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:43.590256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] RBAC: Authorizing `identity:delete_user(user_id=121b6839e5f64256a3342961494cade5)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:43.593700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:43.678342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '121b6839e5f64256a3342961494cade5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:43.690785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '121b6839e5f64256a3342961494cade5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:43.705266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '121b6839e5f64256a3342961494cade5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:43.707137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '121b6839e5f64256a3342961494cade5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:43.707695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-4b084dac-3c81-4965-9ea7-4ac921c0b49f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '121b6839e5f64256a3342961494cade5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:43.739228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1199/2380] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:43 2022] DELETE /identity/v3/users/121b6839e5f64256a3342961494cade5 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:43.741001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:43.741471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:43.742134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:43.763114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:43.764930 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:43.806174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:43.806469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:43.806750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:43.827904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] MFA Rules not processed for user `c0b6f4cfede44b2aa8f4a8dfc1e211d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:43.829178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:43.841452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1182/2381] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:43.851781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:43.853219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:43.870468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:43.891394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1200/2382] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:43.914544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3e930-3ce1-4ba2-81ae-aa56995da433 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:43.918225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:43.918954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:43.919454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:43.919694 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:43.929800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:43.931362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:43.942213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1183/2383] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:44.317382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:44.319011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:44.333987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ac90e-d240-4b0a-ace5-14911e16366e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:44.338913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:44.339908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:44.340262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:44.340604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] PATH_INFO: `/v3/projects/b34ccf97aacd4f1193e8fec271d4a082` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:44.342185 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b34ccf97aacd4f1193e8fec271d4a082)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:44.344214 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:44.403236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b34ccf97aacd4f1193e8fec271d4a082', 'request_id': 'req-a8955c24-4742-4141-ac86-91caf807974f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.410422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b34ccf97aacd4f1193e8fec271d4a082', 'request_id': 'req-a8955c24-4742-4141-ac86-91caf807974f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.419418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b34ccf97aacd4f1193e8fec271d4a082', 'request_id': 'req-a8955c24-4742-4141-ac86-91caf807974f'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.434844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] The token cache is being invalidate because project b34ccf97aacd4f1193e8fec271d4a082 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:44.435147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.436556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.438624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1201/2384] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:44 2022] DELETE /identity/v3/projects/b34ccf97aacd4f1193e8fec271d4a082 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:44.443446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:44.455415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0558e9-d654-494a-8b52-6540946fd7c7 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:44.535703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:44.536731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:44.537060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:44.537410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] PATH_INFO: `/v3/users/f5171215deb946108a78ed7c59948633` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:44.538775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5171215deb946108a78ed7c59948633)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:44.540789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:44.547359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:44.604526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5171215deb946108a78ed7c59948633'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.612564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5171215deb946108a78ed7c59948633'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.620644 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5171215deb946108a78ed7c59948633'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.636147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5171215deb946108a78ed7c59948633'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.636994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-014cac36-b04a-4984-b7e9-482116ae9505 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5171215deb946108a78ed7c59948633'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:44.644295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1184/2385] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:44 2022] DELETE /identity/v3/users/f5171215deb946108a78ed7c59948633 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:45.059430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.061296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:45.082412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8955c24-4742-4141-ac86-91caf807974f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.156016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.157046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.157485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.157795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] PATH_INFO: `/v3/projects/7c9d667f2d5a428cb1c46b7a15b56e0e` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.159088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c9d667f2d5a428cb1c46b7a15b56e0e)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.161292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.215566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c9d667f2d5a428cb1c46b7a15b56e0e', 'request_id': 'req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:45.222026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c9d667f2d5a428cb1c46b7a15b56e0e', 'request_id': 'req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:45.233039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c9d667f2d5a428cb1c46b7a15b56e0e', 'request_id': 'req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:45.242672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:45.246656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] The token cache is being invalidate because project 7c9d667f2d5a428cb1c46b7a15b56e0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:45.247329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:45.248570 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f4ab35a4-d8a5-4557-9d90-ab2bf7089268 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:45.252740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1202/2386] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:45 2022] DELETE /identity/v3/projects/7c9d667f2d5a428cb1c46b7a15b56e0e => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:45.256546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.257025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.257214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.274851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:45.276278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:45.349617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1185/2387] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:45.357355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.357730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.358172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.377946 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:45.379398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:45.388075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1203/2388] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:45.392789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.402536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd61ed5c-b32b-4c2b-b630-3644efcd9d3b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.408036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.408716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.408999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.409283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.410012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:10:45.410318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.411904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.428758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1186/2389] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:10:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:10:45.430384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.441649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7afae5-1d6c-4f1d-a3db-96062835a682 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.446393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.447084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.447371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.447647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.448360 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.450323 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.481447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1204/2390] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:10:45.482572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.492848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be8f1b4-e686-4c39-922f-edcc53ce65ba admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.553417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.554819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.555109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.555388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.556116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.558096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.575274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:45.583768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1187/2391] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/users => generated 291 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:45.587775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.598753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03214d7-abdd-49d2-ba4f-9fabaddce056 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.603711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.604368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.604658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.604975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.605703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.607366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.618975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1205/2392] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:45.622687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.633504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988793b-8454-41c3-ae4f-e09f057398d3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.638217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81ef49f-be73-439d-ab89-49f368c6366e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.638910 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81ef49f-be73-439d-ab89-49f368c6366e admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.639205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81ef49f-be73-439d-ab89-49f368c6366e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.639511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81ef49f-be73-439d-ab89-49f368c6366e admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.640243 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81ef49f-be73-439d-ab89-49f368c6366e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.641942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81ef49f-be73-439d-ab89-49f368c6366e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.652495 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1188/2393] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:10:45.657047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.667342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c32108-ca65-4cb0-a356-5896062ff3be admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.673005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.675531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.675531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.675531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] PATH_INFO: `/v3/projects/a1021e30c9184358a8cf3b5cc6de66d4/users/281dd67926c2441fa030e4a5a5059191/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.690140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1021e30c9184358a8cf3b5cc6de66d4, user_id=281dd67926c2441fa030e4a5a5059191, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.692725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.706907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1206/2394] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:45 2022] PUT /identity/v3/projects/a1021e30c9184358a8cf3b5cc6de66d4/users/281dd67926c2441fa030e4a5a5059191/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:45.718309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.718599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.719039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.747522 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] MFA Rules not processed for user `281dd67926c2441fa030e4a5a5059191`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:45.749737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:45.816088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1189/2395] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:45.820454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.880968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22a25c-0152-4896-aa95-34108cd9d219 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.885597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.886246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.886531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.886810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.889986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:45.914883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1207/2396] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/projects => generated 379 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:45.921191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:45.932182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd93acc3-4744-4462-a611-37dd832327f2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:45.987870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:45.988532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:45.988854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:45.989136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:45.989824 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:45.991737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:46.010209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1190/2397] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:45 2022] POST /identity/v3/users => generated 296 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:46.014660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:46.024833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977a4f7-4dd4-4cf1-a740-cc18ba832b93 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:46.035603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:46.036288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:46.036575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:46.036895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:46.037634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:46.039269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:46.050797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1208/2398] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:46 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:46.055514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:46.068527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8929f8-f175-4314-bdc0-d8c91737b2f5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:46.074607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:46.075496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:46.075782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:46.076533 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] PATH_INFO: `/v3/projects/0f59d3cea1634a6d8b864b2d06d43fbc/users/f01ef7a1e4a74860960f35abee3d11ee/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:46.095011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f59d3cea1634a6d8b864b2d06d43fbc, user_id=f01ef7a1e4a74860960f35abee3d11ee, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:46.109465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:46.143652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1191/2399] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:46 2022] PUT /identity/v3/projects/0f59d3cea1634a6d8b864b2d06d43fbc/users/f01ef7a1e4a74860960f35abee3d11ee/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:46.149998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:46.167653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38bfbf6-f1a3-489e-b028-d7d4af268b85 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:46.238841 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465b29c4-c423-4da7-9b75-04aad4ea98ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:46.239573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465b29c4-c423-4da7-9b75-04aad4ea98ca admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:46.239860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465b29c4-c423-4da7-9b75-04aad4ea98ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:46.240134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465b29c4-c423-4da7-9b75-04aad4ea98ca admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:46.240947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465b29c4-c423-4da7-9b75-04aad4ea98ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:46.243586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465b29c4-c423-4da7-9b75-04aad4ea98ca admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:46.256728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1209/2400] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:46 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:46.261224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:46.273780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16d1b4c-e99a-43b1-af54-96142c4a9dc9 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:46.278855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:46.279838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:46.280114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:46.280385 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] PATH_INFO: `/v3/projects/0f59d3cea1634a6d8b864b2d06d43fbc/users/f01ef7a1e4a74860960f35abee3d11ee/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:46.282379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f59d3cea1634a6d8b864b2d06d43fbc, user_id=f01ef7a1e4a74860960f35abee3d11ee, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:46.284998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:46.300126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1192/2401] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:46 2022] PUT /identity/v3/projects/0f59d3cea1634a6d8b864b2d06d43fbc/users/f01ef7a1e4a74860960f35abee3d11ee/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:46.311398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:46.311691 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:46.311969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:46.329763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:46.344978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] MFA Rules not processed for user `f01ef7a1e4a74860960f35abee3d11ee`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:46.346206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:46.416923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1210/2402] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:46 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:46.427787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:46.429513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:46.469874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:46.497593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f0aa11-876a-4354-bab4-cb1aabf35a34 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:46.501958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7402c26-f3ba-4f0f-903f-8c3210fee6f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:46.502609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7402c26-f3ba-4f0f-903f-8c3210fee6f4 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:46.502896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7402c26-f3ba-4f0f-903f-8c3210fee6f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:46.503168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7402c26-f3ba-4f0f-903f-8c3210fee6f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:46.512954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7402c26-f3ba-4f0f-903f-8c3210fee6f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:46.514945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7402c26-f3ba-4f0f-903f-8c3210fee6f4 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:46.525185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1193/2403] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:46 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:46.965399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:47.028504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555448f-1817-4eff-a8e9-13137820f79a None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:47.033836 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:47.034784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.035173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.035461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] PATH_INFO: `/v3/users/281dd67926c2441fa030e4a5a5059191` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.036710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] RBAC: Authorizing `identity:delete_user(user_id=281dd67926c2441fa030e4a5a5059191)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:47.038672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:47.085659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '281dd67926c2441fa030e4a5a5059191'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.094487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '281dd67926c2441fa030e4a5a5059191'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.101590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '281dd67926c2441fa030e4a5a5059191'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.102228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '281dd67926c2441fa030e4a5a5059191'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.102944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '281dd67926c2441fa030e4a5a5059191'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.120915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1211/2404] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:46 2022] DELETE /identity/v3/users/281dd67926c2441fa030e4a5a5059191 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:47.125595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be928474-53cc-4eee-bfcc-37489bdb2300 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.125883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be928474-53cc-4eee-bfcc-37489bdb2300 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.126159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be928474-53cc-4eee-bfcc-37489bdb2300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.143707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-be928474-53cc-4eee-bfcc-37489bdb2300 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:47.145482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-be928474-53cc-4eee-bfcc-37489bdb2300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:47.216767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1194/2405] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:47.226438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:47.228173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:47.246471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:47.274469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e783b333-0953-4fad-9911-583641632285 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.274950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e783b333-0953-4fad-9911-583641632285 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.275054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e783b333-0953-4fad-9911-583641632285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.293600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e783b333-0953-4fad-9911-583641632285 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:47.295332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e783b333-0953-4fad-9911-583641632285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:47.313492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d65feb-da20-4481-b048-9fc8f84cf642 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:47.318063 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:47.319035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.319346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.319722 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.339924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:47.342689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:47.356624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1212/2406] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:47.387092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1195/2407] 10.222.0.33 () {58 vars in 967 bytes} [Tue Jul 19 11:10:47 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:47.449206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:47.450668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:47.538695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a4416-e13b-4f81-8105-b900041301fd service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:47.547788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:47.548750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.549223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.549683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.561480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:47.563315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:47.573407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1213/2408] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:10:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 19 11:10:47.690236 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e783b333-0953-4fad-9911-583641632285 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:47.691704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e783b333-0953-4fad-9911-583641632285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:47.718464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e783b333-0953-4fad-9911-583641632285 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:47.725475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:47.726631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.726875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.727149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] PATH_INFO: `/v3/projects/a1021e30c9184358a8cf3b5cc6de66d4` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.728789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1021e30c9184358a8cf3b5cc6de66d4)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:47.731064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:47.790873 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1021e30c9184358a8cf3b5cc6de66d4', 'request_id': 'req-5d03ba4f-f117-4a37-956e-77f2d9a0acef'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.797963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1021e30c9184358a8cf3b5cc6de66d4', 'request_id': 'req-5d03ba4f-f117-4a37-956e-77f2d9a0acef'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.808253 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1021e30c9184358a8cf3b5cc6de66d4', 'request_id': 'req-5d03ba4f-f117-4a37-956e-77f2d9a0acef'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.826793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:47.833988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] The token cache is being invalidate because project a1021e30c9184358a8cf3b5cc6de66d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:47.834795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.836609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:47.841798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1196/2409] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:47 2022] DELETE /identity/v3/projects/a1021e30c9184358a8cf3b5cc6de66d4 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:47.846290 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:47.861586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc74ed88-1b4d-47e6-b11d-b15a4ae1a279 service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:47.975381 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:47.977595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:47.977888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:47.978375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] PATH_INFO: `/v3/users/f01ef7a1e4a74860960f35abee3d11ee` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:47.979687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] RBAC: Authorizing `identity:delete_user(user_id=f01ef7a1e4a74860960f35abee3d11ee)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:47.982522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:48.061936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:48.067663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f01ef7a1e4a74860960f35abee3d11ee'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.084644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f01ef7a1e4a74860960f35abee3d11ee'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.109977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f01ef7a1e4a74860960f35abee3d11ee'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.116484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f01ef7a1e4a74860960f35abee3d11ee'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-9c7d8ea2-1a17-48dd-86b7-5b2e71634fff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f01ef7a1e4a74860960f35abee3d11ee'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.155385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1214/2410] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:47 2022] DELETE /identity/v3/users/f01ef7a1e4a74860960f35abee3d11ee => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:48.819707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:48.821792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:48.835100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03ba4f-f117-4a37-956e-77f2d9a0acef admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:48.898341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:48.899206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:48.899483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:48.899888 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] PATH_INFO: `/v3/projects/0f59d3cea1634a6d8b864b2d06d43fbc` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:48.901095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f59d3cea1634a6d8b864b2d06d43fbc)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:48.902985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:48.960271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f59d3cea1634a6d8b864b2d06d43fbc', 'request_id': 'req-6d3076da-128b-4171-90cb-59274a311134'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.966041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f59d3cea1634a6d8b864b2d06d43fbc', 'request_id': 'req-6d3076da-128b-4171-90cb-59274a311134'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.974673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f59d3cea1634a6d8b864b2d06d43fbc', 'request_id': 'req-6d3076da-128b-4171-90cb-59274a311134'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.987103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] The token cache is being invalidate because project 0f59d3cea1634a6d8b864b2d06d43fbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:48.987347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.987915 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6d3076da-128b-4171-90cb-59274a311134 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:48.990028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1197/2411] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:48 2022] DELETE /identity/v3/projects/0f59d3cea1634a6d8b864b2d06d43fbc => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:48.994597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:48.995227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:48.995227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.015146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:49.016357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:49.078584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1215/2412] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:49.087429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.087717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.087997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.105966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:49.107170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:49.115618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1198/2413] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:49.120356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.130869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2fa15-e03b-4a51-8504-a828c10f51cc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.135441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.136091 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.136373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.136645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.137698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:10:49.138006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.139594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.151590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1216/2414] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:10:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:10:49.156382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.165782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717a863f-8e26-4979-afae-25ddd25401e5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.171091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.171748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.172034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.172309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.173044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.174976 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.203979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1199/2415] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:10:49 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:49.205249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.215652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623eeb2e-f61a-4669-abbe-948ddb0c1ba4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.272532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.273229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.273512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.273785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.274493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.276683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.299860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1217/2416] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:10:49 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:10:49.308734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.326116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7ef8a0-5958-4bd6-a55f-8be1fb4ac338 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.331358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.332043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.332327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.332602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.333378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.335066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.347026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1200/2417] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:49 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:10:49.351571 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.364128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc434fda-bf53-477d-9cdb-172adeb0aaf5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.370793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a226fbc-69df-4e40-8ca7-410e55549c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.371786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a226fbc-69df-4e40-8ca7-410e55549c7e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.372669 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a226fbc-69df-4e40-8ca7-410e55549c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.373477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a226fbc-69df-4e40-8ca7-410e55549c7e admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.374690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a226fbc-69df-4e40-8ca7-410e55549c7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.377398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a226fbc-69df-4e40-8ca7-410e55549c7e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.400357 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1218/2418] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:10:49 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:10:49.404684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.416153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0310d989-329a-4684-843f-eaf7b83348bf admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.422542 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.424761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.424761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.424761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] PATH_INFO: `/v3/projects/15be2bc261034dd7a8e86ad15816b579/users/5f7bd8d2bcc44ca3a341537d88bd294a/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.441083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15be2bc261034dd7a8e86ad15816b579, user_id=5f7bd8d2bcc44ca3a341537d88bd294a, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.444637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.458678 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1201/2419] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:10:49 2022] PUT /identity/v3/projects/15be2bc261034dd7a8e86ad15816b579/users/5f7bd8d2bcc44ca3a341537d88bd294a/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:49.469665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d9db97-7fea-438f-8218-b3a970e86dc7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.469953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d9db97-7fea-438f-8218-b3a970e86dc7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.470236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d9db97-7fea-438f-8218-b3a970e86dc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.501922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-49d9db97-7fea-438f-8218-b3a970e86dc7 None None] MFA Rules not processed for user `5f7bd8d2bcc44ca3a341537d88bd294a`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:49.503209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-49d9db97-7fea-438f-8218-b3a970e86dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:49.524918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:49.585440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1219/2420] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:49.602614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:49.604327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:49.647965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:49.694551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7605ba-2116-437e-92b0-d725dbb4f634 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:49.699599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:49.701861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:49.702283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:49.702666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:49.713411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:49.715155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:49.726765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1202/2421] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:55.470604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:55.471422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:55.471862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:55.498735 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:55.500511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:55.617689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1220/2422] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:10:55 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:10:55.639838 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:55.641753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:55.740543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9ff911-ed25-47b3-807c-642573cd3eaf service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:55.748745 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:55.750576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:55.750878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:55.751161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:55.765395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:55.767207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:55.784147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1203/2423] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:10:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 19 11:10:56.471066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:56.472572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:56.567901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:56.569710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:56.608035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5d0124-ee92-4267-9213-f03ffe35f853 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:56.620324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:56.621044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:56.621324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:56.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:56.623692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:56.637022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:56.639187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:56.651791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1221/2424] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:56 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:10:56.688400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f969c789-ef52-4b2f-b3f1-88e402ad9cd8 service swift] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:56.702212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:56.703812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:56.704380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:56.704791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] PATH_INFO: `/v3/users/9e853b49c2c744e89243ec51e7a1a379` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:56.706673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e853b49c2c744e89243ec51e7a1a379)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:56.710509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:56.801107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e853b49c2c744e89243ec51e7a1a379'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:56.815601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e853b49c2c744e89243ec51e7a1a379'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:56.822547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:56.827223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e853b49c2c744e89243ec51e7a1a379'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:56.859059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e853b49c2c744e89243ec51e7a1a379'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:56.859882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e853b49c2c744e89243ec51e7a1a379'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:56.870424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1204/2425] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:56 2022] DELETE /identity/v3/users/9e853b49c2c744e89243ec51e7a1a379 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:57.199569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:57.217037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cde2495-36b8-4a85-bafd-c30eb1c71013 service placement] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:57.331145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:57.367039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:57.367944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:57.368245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:57.368550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] PATH_INFO: `/v3/projects/05630eeaf84a4d1bafd213d419f7a97a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:57.369906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=05630eeaf84a4d1bafd213d419f7a97a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:57.371839 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:57.473227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05630eeaf84a4d1bafd213d419f7a97a', 'request_id': 'req-be019445-396a-459c-998e-c5cfb19c3ea5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.483110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05630eeaf84a4d1bafd213d419f7a97a', 'request_id': 'req-be019445-396a-459c-998e-c5cfb19c3ea5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.500055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05630eeaf84a4d1bafd213d419f7a97a', 'request_id': 'req-be019445-396a-459c-998e-c5cfb19c3ea5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.523997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] The token cache is being invalidate because project 05630eeaf84a4d1bafd213d419f7a97a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:57.524976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.528155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.534368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1222/2426] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:57 2022] DELETE /identity/v3/projects/05630eeaf84a4d1bafd213d419f7a97a => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:57.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:57.536959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:57.556528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a4cb0-4b48-4ed6-bbc0-885dfca258ac admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:57.653536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:57.663772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:57.663772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:57.663772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] PATH_INFO: `/v3/users/4162e7aaa48d492b84ad9a44e5b9aad5` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:57.671049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] RBAC: Authorizing `identity:delete_user(user_id=4162e7aaa48d492b84ad9a44e5b9aad5)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:57.676267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:57.743601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4162e7aaa48d492b84ad9a44e5b9aad5'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.756024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4162e7aaa48d492b84ad9a44e5b9aad5'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.776516 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4162e7aaa48d492b84ad9a44e5b9aad5'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.797028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4162e7aaa48d492b84ad9a44e5b9aad5'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.798600 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4162e7aaa48d492b84ad9a44e5b9aad5'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:57.807963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1205/2427] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:57 2022] DELETE /identity/v3/users/4162e7aaa48d492b84ad9a44e5b9aad5 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:10:58.024012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:58.025837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:58.042320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be019445-396a-459c-998e-c5cfb19c3ea5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:58.127508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:58.129444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:58.130010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:58.130304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] PATH_INFO: `/v3/projects/fc778b6ab5134315899752e000cc5989` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:58.131732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc778b6ab5134315899752e000cc5989)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:58.134680 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:58.189124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc778b6ab5134315899752e000cc5989', 'request_id': 'req-2a200fa0-0715-47c9-80af-df4539475625'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:58.196435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc778b6ab5134315899752e000cc5989', 'request_id': 'req-2a200fa0-0715-47c9-80af-df4539475625'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:58.205739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc778b6ab5134315899752e000cc5989', 'request_id': 'req-2a200fa0-0715-47c9-80af-df4539475625'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:58.221228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] The token cache is being invalidate because project fc778b6ab5134315899752e000cc5989 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:58.221228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:58.221228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-2a200fa0-0715-47c9-80af-df4539475625 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:58.223250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1223/2428] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:58 2022] DELETE /identity/v3/projects/fc778b6ab5134315899752e000cc5989 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:58.854556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:58.856882 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:58.873881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186f03e-82eb-4b16-ba6e-7b5874430dce admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:58.961133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:58.973752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:58.974872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:58.975262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:58.975673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] PATH_INFO: `/v3/users/5f7bd8d2bcc44ca3a341537d88bd294a` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:58.977052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f7bd8d2bcc44ca3a341537d88bd294a)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:58.981201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:59.091146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f7bd8d2bcc44ca3a341537d88bd294a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.101949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f7bd8d2bcc44ca3a341537d88bd294a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.109942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f7bd8d2bcc44ca3a341537d88bd294a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.123408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f7bd8d2bcc44ca3a341537d88bd294a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.124241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f7bd8d2bcc44ca3a341537d88bd294a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.130750 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1206/2429] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:10:58 2022] DELETE /identity/v3/users/5f7bd8d2bcc44ca3a341537d88bd294a => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:59.135565 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:59.136065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:59.136578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:59.159742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:59.161274 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:59.170758 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:10:59.235643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1224/2430] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:10:59.245573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:59.247118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:59.332115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86614853-1a63-465e-8819-7b6e590d7b81 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:59.347928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c605f8a2-5376-459d-b9ff-cbde4931224e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:59.347928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c605f8a2-5376-459d-b9ff-cbde4931224e service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:59.347928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c605f8a2-5376-459d-b9ff-cbde4931224e service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:59.347928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c605f8a2-5376-459d-b9ff-cbde4931224e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:59.359207 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c605f8a2-5376-459d-b9ff-cbde4931224e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:59.360848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c605f8a2-5376-459d-b9ff-cbde4931224e service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:59.372793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1207/2431] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:10:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:10:59.678104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:10:59.679977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:10:59.711279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2e59aa-29da-4da4-93ac-eef7f9e4586d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:10:59.730830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:10:59.731787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:59.732128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:59.732471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] PATH_INFO: `/v3/projects/15be2bc261034dd7a8e86ad15816b579` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:59.733833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] RBAC: Authorizing `identity:delete_project(project_id=15be2bc261034dd7a8e86ad15816b579)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:10:59.735756 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:10:59.848955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15be2bc261034dd7a8e86ad15816b579', 'request_id': 'req-06e0b154-506e-4695-a898-de1f1a7fb13b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.862533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15be2bc261034dd7a8e86ad15816b579', 'request_id': 'req-06e0b154-506e-4695-a898-de1f1a7fb13b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.871651 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15be2bc261034dd7a8e86ad15816b579', 'request_id': 'req-06e0b154-506e-4695-a898-de1f1a7fb13b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.901113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] The token cache is being invalidate because project 15be2bc261034dd7a8e86ad15816b579 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:10:59.901435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.905096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-06e0b154-506e-4695-a898-de1f1a7fb13b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:10:59.907244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1225/2432] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:10:59 2022] DELETE /identity/v3/projects/15be2bc261034dd7a8e86ad15816b579 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:10:59.911944 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:10:59.912439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:10:59.912540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:10:59.947464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:10:59.948686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:00.027686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.028115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.028494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.035219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1208/2433] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:10:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:00.054181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:00.056657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:00.068204 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1226/2434] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:00.071808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.091480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe6cd5f-90bd-40a5-b5f2-31fa6f28a85c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.096222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.096917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.097212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.097478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.098214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:11:00.098565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.102150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.118125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.124503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1209/2435] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:11:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:11:00.131298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce88831-945f-4e24-9860-3c3e4fa855ae None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.143229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.144741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.145232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.145729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.146872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.149026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.187797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1227/2436] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:11:00 2022] POST /identity/v3/projects => generated 375 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:11:00.192050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.207581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df130aa5-a694-4a5e-a3ec-b72be0bf1fae admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.211602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:00.293140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.293911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.295269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.295269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.296078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.299664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.327775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1210/2437] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:11:00 2022] POST /identity/v3/users => generated 288 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:11:00.329304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.341403 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05343fa8-3da0-4cc5-ae1f-3590417c4ff9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.346668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.347465 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.347962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.348231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.349625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.351336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.364266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1228/2438] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:11:00.367883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.381043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e574fd2-9662-40b8-aebf-e3b5b5ee5b4d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.385799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcd29a4-724f-46c1-9538-5b218ec6a1f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.386818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcd29a4-724f-46c1-9538-5b218ec6a1f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.386818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcd29a4-724f-46c1-9538-5b218ec6a1f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.387025 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcd29a4-724f-46c1-9538-5b218ec6a1f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.387731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcd29a4-724f-46c1-9538-5b218ec6a1f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.389404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcd29a4-724f-46c1-9538-5b218ec6a1f0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.404890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1211/2439] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:00.406543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.422611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7386a784-d1d6-4c0b-8334-31becccd6675 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.429381 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.430905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.431652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.432133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] PATH_INFO: `/v3/projects/ec198a670f7f4ee1811077e66892fb77/users/371a4bbfc9e74ef3ae624a44451e324b/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.452908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec198a670f7f4ee1811077e66892fb77, user_id=371a4bbfc9e74ef3ae624a44451e324b, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.456130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1229/2440] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:11:00 2022] PUT /identity/v3/projects/ec198a670f7f4ee1811077e66892fb77/users/371a4bbfc9e74ef3ae624a44451e324b/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:00.547490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.548173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.548656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.581957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] MFA Rules not processed for user `371a4bbfc9e74ef3ae624a44451e324b`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:00.583532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:00.677491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1212/2441] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:00 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:00.682823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.721216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:00.758491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba1c14d-83c8-411b-b032-b77213efebe4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.763911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.767418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.767804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.768167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.768982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.770990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.803080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1230/2442] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:11:00 2022] POST /identity/v3/projects => generated 377 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:11:00.808383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.824500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee149b-321e-476c-b006-c4270f687f7d None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.889937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.890816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.891134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.891389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.892256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.894373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.923194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1213/2443] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:11:00 2022] POST /identity/v3/users => generated 289 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:11:00.928885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.952169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ee2cc0-c659-4181-8f26-73fa93cbe0a4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:00.957978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:00.958649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:00.958928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:00.959200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:00.959928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:00.961563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:00.976187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1231/2444] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:00 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:00.979878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:00.991518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:01.001528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6526208-ba08-4f84-8f7f-b33e01ec6297 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.009270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ffb8fc-6665-406c-a167-c7279f4b8147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.009949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ffb8fc-6665-406c-a167-c7279f4b8147 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.010234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ffb8fc-6665-406c-a167-c7279f4b8147 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ffb8fc-6665-406c-a167-c7279f4b8147 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.011359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ffb8fc-6665-406c-a167-c7279f4b8147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.012993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ffb8fc-6665-406c-a167-c7279f4b8147 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.027498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1214/2445] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:00 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:11:01.028765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.039916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c96a2f0-a801-47ad-ba9a-da6cfe5afd60 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.049003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.050426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.050710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.050992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] PATH_INFO: `/v3/projects/89901ead6276428e96bf739c4f2c1e07/users/c6cefe425b74460aaf55b671b89356a9/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.076668 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89901ead6276428e96bf739c4f2c1e07, user_id=c6cefe425b74460aaf55b671b89356a9, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.079445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.108781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1232/2446] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:11:01 2022] PUT /identity/v3/projects/89901ead6276428e96bf739c4f2c1e07/users/c6cefe425b74460aaf55b671b89356a9/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:01.110557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.110852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.111133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.174831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] MFA Rules not processed for user `c6cefe425b74460aaf55b671b89356a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:01.176074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:01.288413 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1215/2447] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:01 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:01.289641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.390533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4020e6-c86d-4d4e-a227-a1f0b0ade2a9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.396442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.397523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.398050 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.398538 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.399524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.401978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.436235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1233/2448] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:11:01 2022] POST /identity/v3/projects => generated 375 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:11:01.437999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.451080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ffbf39-fcec-49ca-a648-7c56b728b2c2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.541386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.542551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.542940 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.543320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.545301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.548201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.582094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1216/2449] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:11:01 2022] POST /identity/v3/users => generated 294 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:11:01.587813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.602327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fb5eb3-952d-45a3-bfcd-9647623f6131 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.609043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.610548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.611243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.611726 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.612679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.614639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.628023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1234/2450] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:01 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:01.632905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.645846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ddd9ca-a636-413e-9524-39174c0de964 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.661702 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.663216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.663733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.664230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] PATH_INFO: `/v3/projects/5fb24888cee74745be47b0e3e1cf5245/users/5ecb59f65a634091b67e9ec611a73941/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.689573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fb24888cee74745be47b0e3e1cf5245, user_id=5ecb59f65a634091b67e9ec611a73941, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.693467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.725294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1217/2451] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:11:01 2022] PUT /identity/v3/projects/5fb24888cee74745be47b0e3e1cf5245/users/5ecb59f65a634091b67e9ec611a73941/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:01.729910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.748627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a339e-b6e6-4084-8c54-946805ab56c1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.859837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:01.891129 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a31c0f-112d-4242-8cf4-f239290bc022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.892007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a31c0f-112d-4242-8cf4-f239290bc022 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a31c0f-112d-4242-8cf4-f239290bc022 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.892575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a31c0f-112d-4242-8cf4-f239290bc022 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.893639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a31c0f-112d-4242-8cf4-f239290bc022 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.896377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a31c0f-112d-4242-8cf4-f239290bc022 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:01.937629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1235/2452] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:01 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:01.940979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:01.964595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c38f7d-ea34-4407-a0fa-8db4ecdfbe7b admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:01.978629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:01.979543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:01.979949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:01.980163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] PATH_INFO: `/v3/projects/5fb24888cee74745be47b0e3e1cf5245/users/5ecb59f65a634091b67e9ec611a73941/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:01.982743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fb24888cee74745be47b0e3e1cf5245, user_id=5ecb59f65a634091b67e9ec611a73941, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:01.985428 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:02.011059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1218/2453] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:11:01 2022] PUT /identity/v3/projects/5fb24888cee74745be47b0e3e1cf5245/users/5ecb59f65a634091b67e9ec611a73941/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:02.022806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da053f42-6e04-4a81-a82e-8282a9545e07 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:02.023099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da053f42-6e04-4a81-a82e-8282a9545e07 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:02.023376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da053f42-6e04-4a81-a82e-8282a9545e07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:02.075974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-da053f42-6e04-4a81-a82e-8282a9545e07 None None] MFA Rules not processed for user `5ecb59f65a634091b67e9ec611a73941`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:02.077861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-da053f42-6e04-4a81-a82e-8282a9545e07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:02.223923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1236/2454] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:02 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:02.240501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:02.242637 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:02.331151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:02.375612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff26ff4f-0161-47b5-aff6-b97b0642ef81 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:02.382681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:02.383453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:02.383816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:02.384174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:02.519310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:02.520684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:02.545064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1219/2455] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:02 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:04.152041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:04.152509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:04.152607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:04.171713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] MFA Rules not processed for user `a217e983effa43f3a4f84221984189c5`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:04.173155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:04.249840 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1237/2456] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:04 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:04.521496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:04.533418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419d45a-1318-4453-80d1-43388cca2608 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:04.537319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:04.538039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:04.538316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:04.538596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:04.647163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:04.648602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:04.670596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1220/2457] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:04 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:04.705345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:04.706876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:04.785558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b0582-2aad-451a-b3dd-cca90b069cfe None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:04.790821 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:04.791989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:04.792787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:04.793081 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:04.798120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:04.814040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:04.815768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:04.829105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1238/2458] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:04 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:05.413767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:05.424764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb895d-16cc-42b7-a94f-449b38c7d99f service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:05.428758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f32e45e-c0b5-4c28-869b-4c960febc4e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:05.429439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f32e45e-c0b5-4c28-869b-4c960febc4e3 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:05.429877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f32e45e-c0b5-4c28-869b-4c960febc4e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:05.430185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f32e45e-c0b5-4c28-869b-4c960febc4e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:05.441295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f32e45e-c0b5-4c28-869b-4c960febc4e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:05.442720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f32e45e-c0b5-4c28-869b-4c960febc4e3 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:05.456728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1221/2459] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:05 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:09.944287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:09.974362 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:10.010173 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd61525-23e0-4a62-a449-60da3d6aefca service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:10.014728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:10.015590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:10.015876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:10.016158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] PATH_INFO: `/v3/users/371a4bbfc9e74ef3ae624a44451e324b` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:10.017355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] RBAC: Authorizing `identity:delete_user(user_id=371a4bbfc9e74ef3ae624a44451e324b)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:10.019276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:10.063018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '371a4bbfc9e74ef3ae624a44451e324b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.069801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '371a4bbfc9e74ef3ae624a44451e324b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.077220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '371a4bbfc9e74ef3ae624a44451e324b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.077853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '371a4bbfc9e74ef3ae624a44451e324b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.078449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '371a4bbfc9e74ef3ae624a44451e324b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.095770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1239/2460] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:11:09 2022] DELETE /identity/v3/users/371a4bbfc9e74ef3ae624a44451e324b => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:10.100143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:10.100583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:10.100751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:10.118066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:10.119251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:10.182806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1222/2461] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:10.192559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:10.193945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:10.255650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b980f2-af57-40ff-afec-91091ab98505 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:10.259301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:10.259938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:10.260219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:10.260491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:10.269627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:10.271035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:10.281077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1240/2462] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:10.542846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:10.544360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:10.559894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25235e05-a6e1-4255-9e96-6425dd0dd19c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:10.565309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:10.566306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:10.566596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:10.566869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] PATH_INFO: `/v3/projects/ec198a670f7f4ee1811077e66892fb77` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:10.568187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec198a670f7f4ee1811077e66892fb77)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:10.570630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:10.582332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:10.622052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec198a670f7f4ee1811077e66892fb77', 'request_id': 'req-7433fd77-3a2a-40d3-b73f-71caee452e64'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.628573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec198a670f7f4ee1811077e66892fb77', 'request_id': 'req-7433fd77-3a2a-40d3-b73f-71caee452e64'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.636369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec198a670f7f4ee1811077e66892fb77', 'request_id': 'req-7433fd77-3a2a-40d3-b73f-71caee452e64'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.648947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] The token cache is being invalidate because project ec198a670f7f4ee1811077e66892fb77 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:11:10.649851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.650504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.652376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1223/2463] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:11:10 2022] DELETE /identity/v3/projects/ec198a670f7f4ee1811077e66892fb77 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:10.656832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:10.667315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88988690-de04-4819-9d7c-e4cbc56eb132 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:10.715435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:10.729645 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:10.730425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:10.730711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:10.730991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] PATH_INFO: `/v3/users/c6cefe425b74460aaf55b671b89356a9` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:10.732193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6cefe425b74460aaf55b671b89356a9)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:10.734146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:10.781442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6cefe425b74460aaf55b671b89356a9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.789283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6cefe425b74460aaf55b671b89356a9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.796139 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6cefe425b74460aaf55b671b89356a9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.796791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6cefe425b74460aaf55b671b89356a9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.797402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6cefe425b74460aaf55b671b89356a9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:10.814134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1241/2464] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:11:10 2022] DELETE /identity/v3/users/c6cefe425b74460aaf55b671b89356a9 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:11.089208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:11.090726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:11.103666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433fd77-3a2a-40d3-b73f-71caee452e64 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:11.159743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:11.160614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:11.160934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:11.161213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] PATH_INFO: `/v3/projects/89901ead6276428e96bf739c4f2c1e07` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:11.162421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] RBAC: Authorizing `identity:delete_project(project_id=89901ead6276428e96bf739c4f2c1e07)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:11.164351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:11.208754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89901ead6276428e96bf739c4f2c1e07', 'request_id': 'req-e5f2ce32-cfbb-495a-97c0-c68414180807'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.214204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89901ead6276428e96bf739c4f2c1e07', 'request_id': 'req-e5f2ce32-cfbb-495a-97c0-c68414180807'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.221779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89901ead6276428e96bf739c4f2c1e07', 'request_id': 'req-e5f2ce32-cfbb-495a-97c0-c68414180807'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.233517 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] The token cache is being invalidate because project 89901ead6276428e96bf739c4f2c1e07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:11:11.233806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.234446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.235983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1224/2465] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:11:11 2022] DELETE /identity/v3/projects/89901ead6276428e96bf739c4f2c1e07 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:11.239978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:11.241442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:11.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e2225-ba85-431f-9167-c843af77d383 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:11.310557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:11.311447 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:11.311731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:11.312015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] PATH_INFO: `/v3/users/5ecb59f65a634091b67e9ec611a73941` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:11.313240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ecb59f65a634091b67e9ec611a73941)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:11.315163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:11.363145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ecb59f65a634091b67e9ec611a73941'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.371098 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ecb59f65a634091b67e9ec611a73941'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.379297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ecb59f65a634091b67e9ec611a73941'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.379948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ecb59f65a634091b67e9ec611a73941'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.380685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-700ae8f3-fa03-4fcf-9e7b-220941b79ed8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ecb59f65a634091b67e9ec611a73941'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.386034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:11.401436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1242/2466] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:11:11 2022] DELETE /identity/v3/users/5ecb59f65a634091b67e9ec611a73941 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:11.673503 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:11.675006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:11.690355 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f2ce32-cfbb-495a-97c0-c68414180807 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:11.719019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:11.755939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:11.756893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:11.757172 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:11.757554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] PATH_INFO: `/v3/projects/5fb24888cee74745be47b0e3e1cf5245` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:11.768571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fb24888cee74745be47b0e3e1cf5245)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:11.770583 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:11.844506 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fb24888cee74745be47b0e3e1cf5245', 'request_id': 'req-fe5abf48-9a16-4abc-850b-c3abbe5959e4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.852451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fb24888cee74745be47b0e3e1cf5245', 'request_id': 'req-fe5abf48-9a16-4abc-850b-c3abbe5959e4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.862309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fb24888cee74745be47b0e3e1cf5245', 'request_id': 'req-fe5abf48-9a16-4abc-850b-c3abbe5959e4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.881047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] The token cache is being invalidate because project 5fb24888cee74745be47b0e3e1cf5245 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:11:11.881354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.882444 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fe5abf48-9a16-4abc-850b-c3abbe5959e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:11.884078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1225/2467] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:11:11 2022] DELETE /identity/v3/projects/5fb24888cee74745be47b0e3e1cf5245 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:11.888945 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:11.889247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:11.889532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:11.910760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:11.912080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:11.986916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1243/2468] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:11.994964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:11.995284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:11.995566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.016489 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:12.017779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:12.025934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1226/2469] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:11:12.030977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.040879 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db80989-2cd9-4bda-be32-0b9f35464915 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.046021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.046671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.046952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.047224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.047951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:11:12.048280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.049968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.061556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1244/2470] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:11:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:11:12.066019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.075214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee997300-b3f0-428a-b15d-e715979e0005 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.080540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.081222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.081508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.082248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.082454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.084496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.111344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1227/2471] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:11:12 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:11:12.115657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.126055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727acad1-ef1b-4748-8c8b-0e28ad51699e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.180770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.181428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.181710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.181982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.182653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.184563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.204418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1245/2472] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:11:12 2022] POST /identity/v3/users => generated 286 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:11:12.208654 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.222015 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d610c7f3-5f27-4272-ab01-d611a9df90bf admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.226706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.227367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.227660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.227917 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.228675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.230280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.242553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1228/2473] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:11:12.246473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.257231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b28cc2-8882-4dfd-a99a-fcf47c2477ca admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.262567 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e7ae6c-c3e5-4620-a157-d6f25dc33921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.266020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e7ae6c-c3e5-4620-a157-d6f25dc33921 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.266989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e7ae6c-c3e5-4620-a157-d6f25dc33921 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.267306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e7ae6c-c3e5-4620-a157-d6f25dc33921 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.268265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e7ae6c-c3e5-4620-a157-d6f25dc33921 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.270086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e7ae6c-c3e5-4620-a157-d6f25dc33921 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.284877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1246/2474] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:12.288192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.298538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0997cb42-e1db-4a3b-a0e6-171250449991 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.303370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.304257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.304566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.304859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] PATH_INFO: `/v3/projects/a80210f7cff640a5abccbdf35dcf90cb/users/f047610ef36d4b2fb29a0daeb54e354c/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.320607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a80210f7cff640a5abccbdf35dcf90cb, user_id=f047610ef36d4b2fb29a0daeb54e354c, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.323320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.337344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1229/2475] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:11:12 2022] PUT /identity/v3/projects/a80210f7cff640a5abccbdf35dcf90cb/users/f047610ef36d4b2fb29a0daeb54e354c/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:12.348194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8344b9ec-dd0c-44c9-a2fc-5da00759fcdf None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.348490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8344b9ec-dd0c-44c9-a2fc-5da00759fcdf None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.348801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8344b9ec-dd0c-44c9-a2fc-5da00759fcdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.383552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8344b9ec-dd0c-44c9-a2fc-5da00759fcdf None None] MFA Rules not processed for user `f047610ef36d4b2fb29a0daeb54e354c`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:12.384803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8344b9ec-dd0c-44c9-a2fc-5da00759fcdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:12.426037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:12.453222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1247/2476] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:12 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:12.464872 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:12.466602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:12.524939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0015d9-3b3c-462a-b31e-4322638baed2 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:12.528673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d86736b-9409-4b9c-af2b-86a0dcbd7c98 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:12.529351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d86736b-9409-4b9c-af2b-86a0dcbd7c98 service glance] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:12.529638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d86736b-9409-4b9c-af2b-86a0dcbd7c98 service glance] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:12.529912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d86736b-9409-4b9c-af2b-86a0dcbd7c98 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:12.538748 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d86736b-9409-4b9c-af2b-86a0dcbd7c98 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:12.540214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d86736b-9409-4b9c-af2b-86a0dcbd7c98 service glance] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:12.543829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:12.551800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1230/2477] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:13.537764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff812a71-8e03-4398-84ae-ba07545aa1fd None None] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:13.538297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff812a71-8e03-4398-84ae-ba07545aa1fd None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:13.538297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff812a71-8e03-4398-84ae-ba07545aa1fd None None] PATH_INFO: `/` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:13.539831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1248/2478] 10.222.0.33 () {54 vars in 921 bytes} [Tue Jul 19 11:11:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 19 11:11:13.545267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4418900-02c0-423a-8644-98211ee5bbc8 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:13.545849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4418900-02c0-423a-8644-98211ee5bbc8 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:13.546038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4418900-02c0-423a-8644-98211ee5bbc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:13.565022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f4418900-02c0-423a-8644-98211ee5bbc8 None None] MFA Rules not processed for user `2a1c13591bab47d7ba589e14b83f969e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:13.566677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f4418900-02c0-423a-8644-98211ee5bbc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:13.642921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1231/2479] 10.222.0.33 () {58 vars in 1018 bytes} [Tue Jul 19 11:11:13 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:11:13.655151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff812a71-8e03-4398-84ae-ba07545aa1fd None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:13.657168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ff812a71-8e03-4398-84ae-ba07545aa1fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:13.670783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff812a71-8e03-4398-84ae-ba07545aa1fd None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:13.674528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:13.675178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:13.675461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:13.675741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:13.687333 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:13.688776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:13.700065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1249/2480] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:13 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:15.586649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:15.587182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:15.587277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:15.606380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:15.608133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:15.662390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1232/2481] 10.222.0.33 () {58 vars in 1009 bytes} [Tue Jul 19 11:11:15 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:15.674795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:15.676189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:15.744309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9869907b-81db-46d9-b4b3-606445ec1795 service glance] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:15.749464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:15.750319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:15.750610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:15.750887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:15.762064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:15.763746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:15.772800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1250/2482] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:11:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:11:25.655810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=57 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:25.769142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=46 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:36.203320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:36.205475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:36.273109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b894782f-82c6-4466-8e1c-bd832fd34fc3 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:36.276661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:36.277337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:36.277620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:36.277893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:36.279696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:36.338765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:36.340184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:36.349193 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1233/2483] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:36 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:39.383461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:39.385009 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:39.401122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae25c41-f1b8-421a-8294-9851bc9eabb5 service swift] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:39.406575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:39.407208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:39.407520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:39.407872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:39.409777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:39.467910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:39.469654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1640054-c42a-46fd-8d97-e1aa73b47723 service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:39.478723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1251/2484] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:39 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:46.345629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=48 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:46.646363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:46.706997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11a5269-0283-482d-a8f3-b5c9b13cf022 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:46.711480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:46.712333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:46.712613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:46.712914 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] PATH_INFO: `/v3/users/f047610ef36d4b2fb29a0daeb54e354c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:46.714076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f047610ef36d4b2fb29a0daeb54e354c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:46.715954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:46.761026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f047610ef36d4b2fb29a0daeb54e354c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:46.769337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f047610ef36d4b2fb29a0daeb54e354c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:46.776818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f047610ef36d4b2fb29a0daeb54e354c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:46.789689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f047610ef36d4b2fb29a0daeb54e354c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:46.790383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f047610ef36d4b2fb29a0daeb54e354c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:46.796587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1234/2485] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:11:46 2022] DELETE /identity/v3/users/f047610ef36d4b2fb29a0daeb54e354c => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:11:46.801082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:46.801561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:46.801658 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:46.823437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:46.824577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:46.902885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1252/2486] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:11:46.913001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:46.914463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:46.974966 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9866b97f-c340-4a8e-8b48-542fc8510970 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:46.978734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769fdbdb-d12e-4d9b-ac74-dcb8d16f5623 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:46.979377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769fdbdb-d12e-4d9b-ac74-dcb8d16f5623 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:46.979659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769fdbdb-d12e-4d9b-ac74-dcb8d16f5623 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:46.979934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769fdbdb-d12e-4d9b-ac74-dcb8d16f5623 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:46.989010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769fdbdb-d12e-4d9b-ac74-dcb8d16f5623 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:46.990416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769fdbdb-d12e-4d9b-ac74-dcb8d16f5623 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.000378 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1235/2487] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:47.277595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.279089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:47.298279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a599bf5e-9657-4854-9a37-5135ed8cac14 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.304250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.305371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.305755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.306133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] PATH_INFO: `/v3/projects/a80210f7cff640a5abccbdf35dcf90cb` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.307605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] RBAC: Authorizing `identity:delete_project(project_id=a80210f7cff640a5abccbdf35dcf90cb)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.309550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.356070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a80210f7cff640a5abccbdf35dcf90cb', 'request_id': 'req-6d0b602d-53da-4c20-98f6-07835d3d86da'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:47.361799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a80210f7cff640a5abccbdf35dcf90cb', 'request_id': 'req-6d0b602d-53da-4c20-98f6-07835d3d86da'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:47.369023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a80210f7cff640a5abccbdf35dcf90cb', 'request_id': 'req-6d0b602d-53da-4c20-98f6-07835d3d86da'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:47.380162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] The token cache is being invalidate because project a80210f7cff640a5abccbdf35dcf90cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:11:47.380439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:47.381065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-6d0b602d-53da-4c20-98f6-07835d3d86da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:11:47.382568 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1253/2488] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:11:47 2022] DELETE /identity/v3/projects/a80210f7cff640a5abccbdf35dcf90cb => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:11:47.387588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.388434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.388896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.406765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:47.408527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:47.424628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:47.468308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1236/2489] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:47.476499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.476811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.477087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.494931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:47.496126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:47.504166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1254/2490] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:47.508280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.518322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1f9b6-61f4-49b1-9f52-4abfe4847d5b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.522984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.523663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.523945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.524221 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.524984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:11:47.525294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.526907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.539095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1237/2491] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:11:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:11:47.543332 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.552453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ce50a9-e890-483e-b8b2-38a1bc9a2273 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.557076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.557711 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.557991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.558262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.559016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.561026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.574199 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:47.588133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1255/2492] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:11:47 2022] POST /identity/v3/projects => generated 399 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:11:47.592059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.603010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1667380c-0426-4c87-ba0d-e35891f7f562 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.660717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.661507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.661798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.662073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.662772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.664719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.683913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1238/2493] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:11:47 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:11:47.687876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.698673 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4026a4b8-8db1-48f3-9bd7-bd4a50ac0cfe admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.703374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.704021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.704349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.704622 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.705399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.706998 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.717231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1256/2494] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:47.721401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.731149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc2e704-a484-4a18-a638-77feb65698fb admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.736463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c85ac0-a3d6-430e-a977-ad3afa2528bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.737152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c85ac0-a3d6-430e-a977-ad3afa2528bf admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.737433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c85ac0-a3d6-430e-a977-ad3afa2528bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.737707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c85ac0-a3d6-430e-a977-ad3afa2528bf admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.738427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c85ac0-a3d6-430e-a977-ad3afa2528bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.739997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c85ac0-a3d6-430e-a977-ad3afa2528bf admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.749854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1239/2495] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:11:47.754237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.764043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753031d-d768-4b7b-a7cc-6a2164a37af2 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.768868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.769929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.770240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.770699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] PATH_INFO: `/v3/projects/2f7ea12fcb6445f1b6bcb5690c7ccb56/users/560509d3ce124d88876416b36ebdfd2c/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.786014 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f7ea12fcb6445f1b6bcb5690c7ccb56, user_id=560509d3ce124d88876416b36ebdfd2c, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:47.789151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:47.802040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1257/2496] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:11:47 2022] PUT /identity/v3/projects/2f7ea12fcb6445f1b6bcb5690c7ccb56/users/560509d3ce124d88876416b36ebdfd2c/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:11:47.812148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb8f7e-42e3-41dc-8d2a-634b6b45ef3a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.812437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb8f7e-42e3-41dc-8d2a-634b6b45ef3a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.812744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb8f7e-42e3-41dc-8d2a-634b6b45ef3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.838650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-3ceb8f7e-42e3-41dc-8d2a-634b6b45ef3a None None] MFA Rules not processed for user `560509d3ce124d88876416b36ebdfd2c`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:11:47.839903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3ceb8f7e-42e3-41dc-8d2a-634b6b45ef3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:47.906384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1240/2497] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:11:47 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:11:47.918029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:11:47.920115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:11:47.984528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683d6dc-caa4-4207-a0d0-aaef71d03139 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:11:47.988923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:11:47.989574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:11:47.989860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:11:47.990515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:11:47.999422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:11:48.001813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:11:48.011601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1258/2498] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:11:47 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:11:57.903429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:11:58.007218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:INSERT=2,keystone:SELECT=43 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:00.900981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:00.901561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:00.901561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:00.928825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] MFA Rules not processed for user `2781978834a04fd4a48b4fb2ae2892d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:00.930653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:01.000420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1241/2499] 10.222.0.33 () {58 vars in 967 bytes} [Tue Jul 19 11:12:00 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:01.042844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:01.044379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:01.133760 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0298d826-cb14-4bc6-99f8-5603acddf47d service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:01.139931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2fc6a-77ba-42f3-b4e0-002ebbb96cd3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '110d74df72174fba99493dedd2e98d08', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:01.140973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2fc6a-77ba-42f3-b4e0-002ebbb96cd3 service swift] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:01.140973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2fc6a-77ba-42f3-b4e0-002ebbb96cd3 service swift] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:01.141399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2fc6a-77ba-42f3-b4e0-002ebbb96cd3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:01.151512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e2fc6a-77ba-42f3-b4e0-002ebbb96cd3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:01.153161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e2fc6a-77ba-42f3-b4e0-002ebbb96cd3 service swift] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:01.162005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1259/2500] 10.222.0.33 () {60 vars in 1384 bytes} [Tue Jul 19 11:12:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 19 11:12:01.195186 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:01.280382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db28e7-fba1-4d73-9e05-0855442a4940 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:01.287323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:01.288310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:01.288595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:01.289108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] PATH_INFO: `/v3/users/560509d3ce124d88876416b36ebdfd2c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:01.290341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=560509d3ce124d88876416b36ebdfd2c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:01.292360 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:01.365016 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '560509d3ce124d88876416b36ebdfd2c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:01.378313 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '560509d3ce124d88876416b36ebdfd2c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:01.390320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '560509d3ce124d88876416b36ebdfd2c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:01.409368 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '560509d3ce124d88876416b36ebdfd2c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:01.410200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '560509d3ce124d88876416b36ebdfd2c'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:01.416909 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1242/2501] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:01 2022] DELETE /identity/v3/users/560509d3ce124d88876416b36ebdfd2c => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:01.423297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:01.423825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:01.424774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:01.448736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:01.450057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:01.517688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1260/2502] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:01.527975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:01.529127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:01.563279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:01.600326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb6bbd-a961-4cdb-bb07-98fb00baafaa admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:01.604346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:01.605107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:01.605463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:01.605810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:01.616592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:01.618829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:01.630324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1243/2503] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:02.063892 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:02.066007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:02.101644 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfda7246-cb08-4dfe-8ffe-ebbee2115ef6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:02.117182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:02.118064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:02.118367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:02.118621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] PATH_INFO: `/v3/projects/2f7ea12fcb6445f1b6bcb5690c7ccb56` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:02.120315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f7ea12fcb6445f1b6bcb5690c7ccb56)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:02.122255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:02.180611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f7ea12fcb6445f1b6bcb5690c7ccb56', 'request_id': 'req-37d5237b-ed76-4263-86eb-f798eee4cca6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:02.186740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f7ea12fcb6445f1b6bcb5690c7ccb56', 'request_id': 'req-37d5237b-ed76-4263-86eb-f798eee4cca6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:02.195398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f7ea12fcb6445f1b6bcb5690c7ccb56', 'request_id': 'req-37d5237b-ed76-4263-86eb-f798eee4cca6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:02.214261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] The token cache is being invalidate because project 2f7ea12fcb6445f1b6bcb5690c7ccb56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:02.214547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:02.215198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:02.216805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1261/2504] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:02 2022] DELETE /identity/v3/projects/2f7ea12fcb6445f1b6bcb5690c7ccb56 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:04.678902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:04.689383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0544578-1fb2-42fe-957f-1569e1f653ca service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:04.744676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:04.745424 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:04.745710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:04.745987 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:04.747846 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:04.831683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:04.833268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:04.846842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1244/2505] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:10.023434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:10.026028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:10.040262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d5237b-ed76-4263-86eb-f798eee4cca6 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:10.044496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:10.045441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:10.045732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:10.046010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:10.047788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:10.066619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:10.068405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:10.076076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:10.096982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1262/2506] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:11.551759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:11.555270 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:11.568554 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72275556-f324-4cd7-a2fa-6ed1216c1faa service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:11.573952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:11.574820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:11.575132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:11.575588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] PATH_INFO: `/v3/users/3ece5061796245a1b2a6bef1d9d6323a` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:11.576815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ece5061796245a1b2a6bef1d9d6323a)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:11.578885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:11.638599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ece5061796245a1b2a6bef1d9d6323a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:11.649753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ece5061796245a1b2a6bef1d9d6323a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:11.663904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ece5061796245a1b2a6bef1d9d6323a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:11.675316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:11.682889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ece5061796245a1b2a6bef1d9d6323a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:11.684401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ece5061796245a1b2a6bef1d9d6323a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:11.693132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1245/2507] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:11 2022] DELETE /identity/v3/users/3ece5061796245a1b2a6bef1d9d6323a => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:11.986599 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:12.001338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711fea47-f2a3-4aed-a49b-248249ce4b0b service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:12.114159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:12.115445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:12.116023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:12.116457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] PATH_INFO: `/v3/projects/375de853371a422f883980296a46ad43` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:12.118177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] RBAC: Authorizing `identity:delete_project(project_id=375de853371a422f883980296a46ad43)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:12.120397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:12.179657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '375de853371a422f883980296a46ad43', 'request_id': 'req-736b63d9-74c9-47b1-8cb2-42f55ce98123'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.186187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '375de853371a422f883980296a46ad43', 'request_id': 'req-736b63d9-74c9-47b1-8cb2-42f55ce98123'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.210472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '375de853371a422f883980296a46ad43', 'request_id': 'req-736b63d9-74c9-47b1-8cb2-42f55ce98123'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.228864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] The token cache is being invalidate because project 375de853371a422f883980296a46ad43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:12.228864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.228864 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.231490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1263/2508] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:11 2022] DELETE /identity/v3/projects/375de853371a422f883980296a46ad43 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:12.236834 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:12.238726 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:12.257618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b1d00f-f781-407a-bb0a-a02a6cc14681 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:12.326467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:12.328626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:12.328942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:12.329280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] PATH_INFO: `/v3/users/1640c380a8314c26a0db18e69b206a39` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:12.330609 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1640c380a8314c26a0db18e69b206a39)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:12.333367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:12.387795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1640c380a8314c26a0db18e69b206a39'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.396664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1640c380a8314c26a0db18e69b206a39'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.409337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1640c380a8314c26a0db18e69b206a39'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.424523 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1640c380a8314c26a0db18e69b206a39'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.425316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1640c380a8314c26a0db18e69b206a39'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.431721 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1246/2509] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:12 2022] DELETE /identity/v3/users/1640c380a8314c26a0db18e69b206a39 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:12.668350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:12.669858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:12.683543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b63d9-74c9-47b1-8cb2-42f55ce98123 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:12.760198 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:12.761171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:12.761397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:12.761676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] PATH_INFO: `/v3/projects/37d05fa2cf3643838d6c6a3139bf3953` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:12.762905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37d05fa2cf3643838d6c6a3139bf3953)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:12.764880 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:12.801074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:12.820143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37d05fa2cf3643838d6c6a3139bf3953', 'request_id': 'req-668d56e0-4ecb-4049-86de-b96b184b7574'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.826223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37d05fa2cf3643838d6c6a3139bf3953', 'request_id': 'req-668d56e0-4ecb-4049-86de-b96b184b7574'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.838414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37d05fa2cf3643838d6c6a3139bf3953', 'request_id': 'req-668d56e0-4ecb-4049-86de-b96b184b7574'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.854328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] The token cache is being invalidate because project 37d05fa2cf3643838d6c6a3139bf3953 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:12.854766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.855497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:12.857347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1264/2510] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:12 2022] DELETE /identity/v3/projects/37d05fa2cf3643838d6c6a3139bf3953 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:13.715130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:13.716553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:13.729565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637a4128-2614-4803-8be9-65a116ed6136 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:13.786890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:13.787762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:13.788043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:13.788317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] PATH_INFO: `/v3/users/b8f2a78f3a3e4b8e8937e3bc39524690` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:13.789539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8f2a78f3a3e4b8e8937e3bc39524690)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:13.791446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:13.802118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:13.839501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8f2a78f3a3e4b8e8937e3bc39524690'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:13.846519 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8f2a78f3a3e4b8e8937e3bc39524690'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:13.859418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8f2a78f3a3e4b8e8937e3bc39524690'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:13.872264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8f2a78f3a3e4b8e8937e3bc39524690'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:13.873009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e67bef10-7399-4a80-82f1-3129413cf100 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8f2a78f3a3e4b8e8937e3bc39524690'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:13.880995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1247/2511] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:13 2022] DELETE /identity/v3/users/b8f2a78f3a3e4b8e8937e3bc39524690 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:14.060270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.061853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:14.076775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d56e0-4ecb-4049-86de-b96b184b7574 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.134402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.135261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.135552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.135826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] PATH_INFO: `/v3/projects/ef534f1d06ea415d98857ec7f9e5fb97` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.137035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef534f1d06ea415d98857ec7f9e5fb97)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.138929 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.185363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef534f1d06ea415d98857ec7f9e5fb97', 'request_id': 'req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:14.191101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef534f1d06ea415d98857ec7f9e5fb97', 'request_id': 'req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:14.199457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef534f1d06ea415d98857ec7f9e5fb97', 'request_id': 'req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:14.211545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] The token cache is being invalidate because project ef534f1d06ea415d98857ec7f9e5fb97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:14.211851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:14.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-200f95f9-a86f-4f4d-a942-a4a1395d9fc7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:14.214031 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1265/2512] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:14 2022] DELETE /identity/v3/projects/ef534f1d06ea415d98857ec7f9e5fb97 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:14.218993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.219314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.219624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.238911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:14.240119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:14.300670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1248/2513] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:14.309429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.309719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.309993 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.327667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:14.328896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:14.336670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1266/2514] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:14.340861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.350920 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9efc9c-6f5f-464e-aa12-6c029aec251b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.355492 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.356134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.356416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.356689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.357441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:14.357746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.359326 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.371210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1249/2515] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:12:14.375583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.384650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f8d37-027c-4865-a6c1-fc49fc454077 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.389324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.389958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.390238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.390510 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.391194 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.393174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.417818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1267/2516] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/projects => generated 403 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:14.421926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.434722 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bf714e-db5c-40cc-a21e-2418c3e92a71 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.489618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.490276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.490559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.490833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.491520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.493463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.512795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1250/2517] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/users => generated 302 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:14.517059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.527497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f520d823-a5da-4417-b1fd-0c909010301e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.531968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.532614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.532930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.533748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.534477 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.536059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.546278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1268/2518] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:14.550288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.556715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:14.560758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d025a4e-9142-4b86-b3c9-86d30acfea55 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.570262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb31a2fe-f925-4756-8be7-7f6e733a0927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.571389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb31a2fe-f925-4756-8be7-7f6e733a0927 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.571932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb31a2fe-f925-4756-8be7-7f6e733a0927 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.572369 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb31a2fe-f925-4756-8be7-7f6e733a0927 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.573565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb31a2fe-f925-4756-8be7-7f6e733a0927 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.575185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb31a2fe-f925-4756-8be7-7f6e733a0927 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.587957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1251/2519] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:14.592305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.605373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35d7be-a568-4cd5-b3a4-80ce001dece5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.609923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.610793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.611383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] PATH_INFO: `/v3/projects/b0d9b8ee7fa64564977bc06735acb337/users/0d303315619c405fad203b9416992762/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.619377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:14.627733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0d9b8ee7fa64564977bc06735acb337, user_id=0d303315619c405fad203b9416992762, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.630607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.643504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1269/2520] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:14 2022] PUT /identity/v3/projects/b0d9b8ee7fa64564977bc06735acb337/users/0d303315619c405fad203b9416992762/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:14.653540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.653831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.654109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.681184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] MFA Rules not processed for user `0d303315619c405fad203b9416992762`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:14.682441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:14.745071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1252/2521] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:14.749461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.813982 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77097413-5c87-4bc4-a0f5-b6a6d31e2293 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.818612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.819262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.819549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.819829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.820795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.822695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.848202 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1270/2522] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/projects => generated 403 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:14.852206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.862841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e410a8-fe3e-4613-b913-1b447ad73585 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.919749 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.920421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.920719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.921007 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.921707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.923612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.943362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1253/2523] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:14 2022] POST /identity/v3/users => generated 308 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:14.947349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.958251 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6e12b9-3ecf-4b7b-b4b9-c64891df9036 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.962769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.963411 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.963695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.963972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:14.964690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:14.966294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:14.976552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1271/2524] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:14.980346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:14.990173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4eb97c-65ef-4066-a0fb-d709950703ca admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:14.994787 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:14.995652 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:14.995935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:14.996230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] PATH_INFO: `/v3/projects/7860f3dec6b74402b27703eb9e9dbaa5/users/870ffffdeff346fb8d92117b028058b1/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.011455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7860f3dec6b74402b27703eb9e9dbaa5, user_id=870ffffdeff346fb8d92117b028058b1, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:15.014183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:15.027708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1254/2525] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:14 2022] PUT /identity/v3/projects/7860f3dec6b74402b27703eb9e9dbaa5/users/870ffffdeff346fb8d92117b028058b1/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:15.031601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:15.041543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032d85bc-e8a2-45d9-918a-ff013fa3f14f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:15.096736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc048820-d98f-4618-aacc-295f1479716b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:15.097384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc048820-d98f-4618-aacc-295f1479716b admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.097666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc048820-d98f-4618-aacc-295f1479716b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.097938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc048820-d98f-4618-aacc-295f1479716b admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.098718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc048820-d98f-4618-aacc-295f1479716b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:15.100224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc048820-d98f-4618-aacc-295f1479716b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:15.109624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1272/2526] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:15 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:15.113656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:15.124050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e510158-f04c-4882-aa70-2c4745cf9dd1 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:15.129530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:15.130432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.130711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.130983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] PATH_INFO: `/v3/projects/7860f3dec6b74402b27703eb9e9dbaa5/users/870ffffdeff346fb8d92117b028058b1/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.132982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=7860f3dec6b74402b27703eb9e9dbaa5, user_id=870ffffdeff346fb8d92117b028058b1, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:15.135590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:15.146293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1255/2527] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:15 2022] PUT /identity/v3/projects/7860f3dec6b74402b27703eb9e9dbaa5/users/870ffffdeff346fb8d92117b028058b1/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:15.156086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.156375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.156652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.182124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] MFA Rules not processed for user `870ffffdeff346fb8d92117b028058b1`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:15.183310 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:15.227309 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:15.251419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1273/2528] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:15 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:15.262314 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:15.264028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:15.303736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:15.321898 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f743f0-e5be-47be-ab3a-b2596ce4e1ec admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:15.325642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49835927-7ab1-4354-bb3a-dc076f4fc9a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:15.326284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49835927-7ab1-4354-bb3a-dc076f4fc9a3 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.326564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49835927-7ab1-4354-bb3a-dc076f4fc9a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.326839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49835927-7ab1-4354-bb3a-dc076f4fc9a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.335710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49835927-7ab1-4354-bb3a-dc076f4fc9a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:15.337171 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49835927-7ab1-4354-bb3a-dc076f4fc9a3 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:15.346397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1256/2529] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:15 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:15.496159 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:15.557586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b002032b-ad94-432a-831c-abb5ac11d6cc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:15.562108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:15.562961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.563242 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.563517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] PATH_INFO: `/v3/users/0d303315619c405fad203b9416992762` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.565334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d303315619c405fad203b9416992762)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:15.567238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:15.612458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d303315619c405fad203b9416992762'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:15.619313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d303315619c405fad203b9416992762'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:15.626511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d303315619c405fad203b9416992762'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:15.627736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d303315619c405fad203b9416992762'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:15.628337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d303315619c405fad203b9416992762'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:15.645130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1274/2530] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:15 2022] DELETE /identity/v3/users/0d303315619c405fad203b9416992762 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:15.649617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.649911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.650191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.668023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:15.669249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:15.726131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1257/2531] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:15.735710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:15.737126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:15.794542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1bbdff-e71c-42a8-8c7d-8a489c2018d7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:15.798189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:15.798850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:15.799130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:15.799402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:15.808523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:15.810006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:15.819907 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1275/2532] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:16.093927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:16.095380 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:16.107699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67661d16-41b7-493c-a91e-7995bf5fe36a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:16.113029 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:16.113902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.114184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.114461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] PATH_INFO: `/v3/projects/b0d9b8ee7fa64564977bc06735acb337` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.115669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0d9b8ee7fa64564977bc06735acb337)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:16.117856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:16.164875 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0d9b8ee7fa64564977bc06735acb337', 'request_id': 'req-c9abd2a7-2980-4289-97d9-1c620a154ca2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.170203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0d9b8ee7fa64564977bc06735acb337', 'request_id': 'req-c9abd2a7-2980-4289-97d9-1c620a154ca2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.178043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0d9b8ee7fa64564977bc06735acb337', 'request_id': 'req-c9abd2a7-2980-4289-97d9-1c620a154ca2'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.188931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] The token cache is being invalidate because project b0d9b8ee7fa64564977bc06735acb337 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:16.189216 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.190508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.192030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1258/2533] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:16 2022] DELETE /identity/v3/projects/b0d9b8ee7fa64564977bc06735acb337 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:16.196180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:16.202211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:16.206261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87908b20-9326-439e-a8e3-a81d4ad95a5c service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:16.262482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:16.263342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.263629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.263903 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] PATH_INFO: `/v3/users/870ffffdeff346fb8d92117b028058b1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.265111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] RBAC: Authorizing `identity:delete_user(user_id=870ffffdeff346fb8d92117b028058b1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:16.267016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:16.315555 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '870ffffdeff346fb8d92117b028058b1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.323026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '870ffffdeff346fb8d92117b028058b1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.330028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '870ffffdeff346fb8d92117b028058b1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.330648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '870ffffdeff346fb8d92117b028058b1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.331246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-402e3193-9465-43bc-8c49-539d5d9bbaaf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '870ffffdeff346fb8d92117b028058b1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.347963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1276/2534] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:16 2022] DELETE /identity/v3/users/870ffffdeff346fb8d92117b028058b1 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:16.613305 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:16.614761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:16.629003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9abd2a7-2980-4289-97d9-1c620a154ca2 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:16.685249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:16.686101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.686383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.686650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] PATH_INFO: `/v3/projects/7860f3dec6b74402b27703eb9e9dbaa5` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.687812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7860f3dec6b74402b27703eb9e9dbaa5)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:16.690289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:16.692539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:16.737455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7860f3dec6b74402b27703eb9e9dbaa5', 'request_id': 'req-6a0f841c-4679-4693-bca5-33dd4e5de99b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.743156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7860f3dec6b74402b27703eb9e9dbaa5', 'request_id': 'req-6a0f841c-4679-4693-bca5-33dd4e5de99b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.750932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7860f3dec6b74402b27703eb9e9dbaa5', 'request_id': 'req-6a0f841c-4679-4693-bca5-33dd4e5de99b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.762331 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] The token cache is being invalidate because project 7860f3dec6b74402b27703eb9e9dbaa5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:16.762620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.763244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-6a0f841c-4679-4693-bca5-33dd4e5de99b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:16.764789 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1259/2535] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:16 2022] DELETE /identity/v3/projects/7860f3dec6b74402b27703eb9e9dbaa5 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:16.770247 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.770601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.770943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.789612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:16.790805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:16.848436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1277/2536] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:16.857340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.857632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.857913 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.876744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:16.878541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:16.886372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1260/2537] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:16.890451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:16.900219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce932534-bb3c-4e1b-ab33-fb2b3a4e9791 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:16.904788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:16.905437 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.905720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.905994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.906714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:16.907023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:16.908604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:16.920140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1278/2538] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:12:16.924911 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:16.934295 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22378af9-4295-404c-bc23-5b1a9ad0c2ae None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:16.939056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:16.940263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:16.940546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:16.940861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:16.941560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:16.943507 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:16.968160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1261/2539] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:16 2022] POST /identity/v3/projects => generated 355 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:16.972269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:16.982062 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7593fdfd-77a7-4c9a-9ae0-fec26dd5d3dd admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:16.990156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:17.039506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.040160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.040439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.040731 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.041420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.043335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.063895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1279/2540] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:16 2022] POST /identity/v3/users => generated 278 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:17.068526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.078636 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69212a38-c4d9-4184-850f-3995a5547a07 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.083242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.083893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.084176 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.084454 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.085217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.086807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.097933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1262/2541] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:17.101662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.112021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dfed2f-fc91-4737-b2c7-b41f5ac6498c admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.116630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407350bf-58dc-42fa-8ae4-b1dd91e4e3c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.117305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407350bf-58dc-42fa-8ae4-b1dd91e4e3c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.117588 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407350bf-58dc-42fa-8ae4-b1dd91e4e3c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.117861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407350bf-58dc-42fa-8ae4-b1dd91e4e3c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.118577 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407350bf-58dc-42fa-8ae4-b1dd91e4e3c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.120212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407350bf-58dc-42fa-8ae4-b1dd91e4e3c1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.130813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1280/2542] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:17.134933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.144657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec554986-6a31-4ed9-8ca0-c2c7a837226e admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.149199 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.150052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.150329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.150605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] PATH_INFO: `/v3/projects/bb37c168a3c54609be7af7522921d78e/users/dc832e8e6b4b424f94d17c13367e64a6/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.165351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb37c168a3c54609be7af7522921d78e, user_id=dc832e8e6b4b424f94d17c13367e64a6, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.167955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.181383 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1263/2543] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:17 2022] PUT /identity/v3/projects/bb37c168a3c54609be7af7522921d78e/users/dc832e8e6b4b424f94d17c13367e64a6/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:17.191397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.191686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.192078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.218104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] MFA Rules not processed for user `dc832e8e6b4b424f94d17c13367e64a6`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:17.219253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:17.282364 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1281/2544] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:17 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:17.287356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.349514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f6445-935a-40c8-8197-e325f5d76b2d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.354806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.355465 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.355746 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.356019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.356887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.358821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.391096 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1264/2545] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:17 2022] POST /identity/v3/projects => generated 357 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:17.392589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.403306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea4d210-cb38-45e0-a392-3fdd482c33a5 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.458103 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.458757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.459035 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.459302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.459979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.461913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.481271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1282/2546] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:17 2022] POST /identity/v3/users => generated 285 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:17.485239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.496024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89126248-20bd-4b07-a287-5a3593c88540 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.500879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.501528 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.501812 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.502089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.502805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.504387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.515117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1265/2547] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:17.519591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.529304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ecb06-8501-429f-866d-bef5bcaf66a9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.534496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.535396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.535687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.535953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] PATH_INFO: `/v3/projects/2017d4617f3d457cbfeb924cc31d5ace/users/6e82148fcac7443f94ec832aa794644e/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.540938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:17.553686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2017d4617f3d457cbfeb924cc31d5ace, user_id=6e82148fcac7443f94ec832aa794644e, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.556328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.568952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1283/2548] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:17 2022] PUT /identity/v3/projects/2017d4617f3d457cbfeb924cc31d5ace/users/6e82148fcac7443f94ec832aa794644e/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:17.572923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.578065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:17.584671 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d7b4d3-a2a8-405c-80df-755486d4f626 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.641174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5636b-b1d8-4db7-aea0-ec3d5cc0b4fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.641831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5636b-b1d8-4db7-aea0-ec3d5cc0b4fe admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.642115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5636b-b1d8-4db7-aea0-ec3d5cc0b4fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.642390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5636b-b1d8-4db7-aea0-ec3d5cc0b4fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.643626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5636b-b1d8-4db7-aea0-ec3d5cc0b4fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.645251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5636b-b1d8-4db7-aea0-ec3d5cc0b4fe admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.655060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1266/2549] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:17.659809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.669843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0772c3-cdfc-4351-9b75-b74ec276f052 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.674454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.675315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.675593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.675861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] PATH_INFO: `/v3/projects/2017d4617f3d457cbfeb924cc31d5ace/users/6e82148fcac7443f94ec832aa794644e/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.677860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2017d4617f3d457cbfeb924cc31d5ace, user_id=6e82148fcac7443f94ec832aa794644e, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.680446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.690378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1284/2550] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:17 2022] PUT /identity/v3/projects/2017d4617f3d457cbfeb924cc31d5ace/users/6e82148fcac7443f94ec832aa794644e/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:17.700391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.700684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.700994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.727646 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] MFA Rules not processed for user `6e82148fcac7443f94ec832aa794644e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:17.728893 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:17.796390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1267/2551] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:17 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:17.807482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.809203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:17.864895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97cabd7-56e2-4b67-abc9-7d02a22a468a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.868453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.869120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.869404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.869675 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:17.879169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:17.880595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:17.889948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1285/2552] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:17 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:17.961900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:17.971466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5388bbc0-67d7-4c45-a942-a595f7dd55c5 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:17.975057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:17.975712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:17.976031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:17.976330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:18.032984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:18.034450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:18.043419 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1268/2553] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:17 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:27.885837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=37,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:28.039403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:31.908485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:31.910192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:32.005659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49809591-8dad-438a-a819-f87fae21ed63 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:32.009337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:32.010033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:32.010370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:32.010670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:32.012713 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:32.126799 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:32.128148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d209b2b-d889-4579-8f83-7a0e1fa860d7 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:32.148064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1286/2554] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:32.867781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:32.896552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7071e1-5444-43d2-b56a-8df0bd38602b service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:32.901107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:32.901954 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:32.902254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:32.902514 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] PATH_INFO: `/v3/users/dc832e8e6b4b424f94d17c13367e64a6` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:32.903705 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc832e8e6b4b424f94d17c13367e64a6)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:32.905651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:32.958956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc832e8e6b4b424f94d17c13367e64a6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:32.966337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc832e8e6b4b424f94d17c13367e64a6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:32.973809 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc832e8e6b4b424f94d17c13367e64a6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:32.986663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc832e8e6b4b424f94d17c13367e64a6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:32.987382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc832e8e6b4b424f94d17c13367e64a6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:32.993414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1269/2555] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:32 2022] DELETE /identity/v3/users/dc832e8e6b4b424f94d17c13367e64a6 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:32.998026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:32.998500 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:32.998715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:33.033796 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:33.035270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:33.057724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:33.104842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1287/2556] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:33.114046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:33.116043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:33.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1221a1e-e96f-4ed1-93fb-dc8a56483984 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:33.186306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:33.187035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:33.187406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:33.187719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:33.198866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:33.200403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:33.211373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1270/2557] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:33.486285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:33.488342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:33.504469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33f89dd-3a0a-443f-a5b4-fdc4ebabac79 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:33.520396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:33.522812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:33.523596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:33.524556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] PATH_INFO: `/v3/projects/bb37c168a3c54609be7af7522921d78e` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:33.537076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb37c168a3c54609be7af7522921d78e)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:33.539145 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:33.594887 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb37c168a3c54609be7af7522921d78e', 'request_id': 'req-8be0dc06-d17c-4a29-aa68-9a981e601721'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.601444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb37c168a3c54609be7af7522921d78e', 'request_id': 'req-8be0dc06-d17c-4a29-aa68-9a981e601721'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.612541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb37c168a3c54609be7af7522921d78e', 'request_id': 'req-8be0dc06-d17c-4a29-aa68-9a981e601721'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.626550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] The token cache is being invalidate because project bb37c168a3c54609be7af7522921d78e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:33.627007 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.627795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.629798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1288/2558] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:33 2022] DELETE /identity/v3/projects/bb37c168a3c54609be7af7522921d78e => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:33.633582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:33.643865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5bb045-ba47-4563-a91e-119e411c7721 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:33.697754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:33.707771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:33.708863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:33.709759 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:33.710153 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] PATH_INFO: `/v3/users/6e82148fcac7443f94ec832aa794644e` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:33.711556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e82148fcac7443f94ec832aa794644e)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:33.714160 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:33.770352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e82148fcac7443f94ec832aa794644e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.781262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e82148fcac7443f94ec832aa794644e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.791342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e82148fcac7443f94ec832aa794644e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.805896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e82148fcac7443f94ec832aa794644e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.806815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d672f246-7567-450c-9ec4-0a5cf5c3721c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e82148fcac7443f94ec832aa794644e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:33.814844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1271/2559] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:33 2022] DELETE /identity/v3/users/6e82148fcac7443f94ec832aa794644e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:34.141458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.143078 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:34.156720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be0dc06-d17c-4a29-aa68-9a981e601721 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.218728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.219526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.219812 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.220213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] PATH_INFO: `/v3/projects/2017d4617f3d457cbfeb924cc31d5ace` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.221376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2017d4617f3d457cbfeb924cc31d5ace)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.223283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.274498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2017d4617f3d457cbfeb924cc31d5ace', 'request_id': 'req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:34.276537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:34.281331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2017d4617f3d457cbfeb924cc31d5ace', 'request_id': 'req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:34.294690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2017d4617f3d457cbfeb924cc31d5ace', 'request_id': 'req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:34.308548 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] The token cache is being invalidate because project 2017d4617f3d457cbfeb924cc31d5ace was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:34.308886 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:34.309537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-c8e19086-8c39-455b-8d56-a2fb2c1aad4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:34.311296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1289/2560] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:34 2022] DELETE /identity/v3/projects/2017d4617f3d457cbfeb924cc31d5ace => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:34.316468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.316980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.317094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.366376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:34.367713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:34.438350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1272/2561] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:34.446924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.447206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.447479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.480688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:34.482426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:34.495111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1290/2562] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:34.496312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.506935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f50ef0-8d66-44b3-827a-2fb672211cb1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.515881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.516543 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.516866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.517178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.517907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:34.518214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.519839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.532315 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1273/2563] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:12:34.537610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.547118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabd5a32-ac80-4606-8c2e-f8236b3521d7 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.552452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.553128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.553412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.553685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.554379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.556277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.585860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1291/2564] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:34 2022] POST /identity/v3/projects => generated 365 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:34.590155 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.601574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd1c72-fb29-48db-b7c0-e59d01c0acf9 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.645273 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:34.667002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.667883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.668010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.668218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.668994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.670962 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.696957 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1274/2565] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:34 2022] POST /identity/v3/users => generated 283 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:34.698620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.710877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68abece7-b8d5-43cd-856f-ef2edb195bb9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.719416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.720686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.721012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.721289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.722046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.723649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.735418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1292/2566] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:34 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:34.740163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.750992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f264064-0f9b-4faa-9f50-0baf5939b448 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.755908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745eec23-6f01-49e3-a04a-c682c759ebec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.756673 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745eec23-6f01-49e3-a04a-c682c759ebec admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.757071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745eec23-6f01-49e3-a04a-c682c759ebec admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.757421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745eec23-6f01-49e3-a04a-c682c759ebec admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.758246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745eec23-6f01-49e3-a04a-c682c759ebec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.759967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745eec23-6f01-49e3-a04a-c682c759ebec admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.771992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1275/2567] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:34.776386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:34.789128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeab3c4-c3a5-4c2a-a7a0-d632b32c7764 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:34.794228 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:34.795137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.795425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.795704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] PATH_INFO: `/v3/projects/bd8151f6db1545caabc71b804f5b2773/users/007d3e3d344f4e8bbd7b5d917210da39/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.812260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd8151f6db1545caabc71b804f5b2773, user_id=007d3e3d344f4e8bbd7b5d917210da39, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:34.814997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:34.838605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1293/2568] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:34 2022] PUT /identity/v3/projects/bd8151f6db1545caabc71b804f5b2773/users/007d3e3d344f4e8bbd7b5d917210da39/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:34.850665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:34.850973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:34.851283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:34.885717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] MFA Rules not processed for user `007d3e3d344f4e8bbd7b5d917210da39`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:34.887045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:34.982415 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1276/2569] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:34 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:34.988000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.083192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3657087-f814-4b0e-b703-4e89c0d3e84a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.096523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.100260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.100762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.101211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.102076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.104315 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.135871 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1294/2570] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:34 2022] POST /identity/v3/projects => generated 361 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:35.140036 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.170837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb9ca9-b0e9-43a6-982f-8d6a9abc0c29 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.240242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.241079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.241445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.241723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.242425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.244348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.270993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1277/2571] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:35 2022] POST /identity/v3/users => generated 287 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:35.275085 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.287648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854baff7-317b-4c0c-9eca-e5496b82e0d6 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.293321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.293926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.294216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.294496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.295237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.296872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.308633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1295/2572] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:35.312982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.323648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81adca33-1274-4cbd-8b18-2ccdbba04e82 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.329112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.330669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.331299 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.331683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] PATH_INFO: `/v3/projects/985b8db18d0d48c2bda656e04a9bb00a/users/395e4efd1c844a479926c9242bf250a5/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.349020 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] RBAC: Authorizing `identity:create_grant(project_id=985b8db18d0d48c2bda656e04a9bb00a, user_id=395e4efd1c844a479926c9242bf250a5, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.352728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.368997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1278/2573] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:35 2022] PUT /identity/v3/projects/985b8db18d0d48c2bda656e04a9bb00a/users/395e4efd1c844a479926c9242bf250a5/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:35.377040 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.388660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e85ecc-515e-4719-9df0-6794aa289c26 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.422670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.425471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.442124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:35.444992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:35.453777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.454448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.455383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.455659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.456387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.458068 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.468620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1296/2574] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:35.472748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.482209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d619eb-a8a1-4f8e-af86-2e5075151e62 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.486941 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125965fe-3eeb-408b-928d-ab91b23f470a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.487803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125965fe-3eeb-408b-928d-ab91b23f470a admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.488089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125965fe-3eeb-408b-928d-ab91b23f470a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.488366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125965fe-3eeb-408b-928d-ab91b23f470a admin admin] PATH_INFO: `/v3/projects/985b8db18d0d48c2bda656e04a9bb00a/users/395e4efd1c844a479926c9242bf250a5/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.490433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125965fe-3eeb-408b-928d-ab91b23f470a admin admin] RBAC: Authorizing `identity:create_grant(project_id=985b8db18d0d48c2bda656e04a9bb00a, user_id=395e4efd1c844a479926c9242bf250a5, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.493041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125965fe-3eeb-408b-928d-ab91b23f470a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.504026 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1297/2575] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:35 2022] PUT /identity/v3/projects/985b8db18d0d48c2bda656e04a9bb00a/users/395e4efd1c844a479926c9242bf250a5/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:35.505398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743f2ba-4d29-4d72-8669-5d3323afa920 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.515000 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.515383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.515664 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.542358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] MFA Rules not processed for user `395e4efd1c844a479926c9242bf250a5`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:35.543586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.562070 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.562935 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.563214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.563484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] PATH_INFO: `/v3/users/c0b6f4cfede44b2aa8f4a8dfc1e211d6` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.565526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0b6f4cfede44b2aa8f4a8dfc1e211d6)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.567525 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.613663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1298/2576] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:35 2022] POST /identity/v3/auth/tokens => generated 3295 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:35.615939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0b6f4cfede44b2aa8f4a8dfc1e211d6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:35.625057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.625227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0b6f4cfede44b2aa8f4a8dfc1e211d6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:35.626806 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.633723 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0b6f4cfede44b2aa8f4a8dfc1e211d6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:35.646311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0b6f4cfede44b2aa8f4a8dfc1e211d6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:35.646998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0b6f4cfede44b2aa8f4a8dfc1e211d6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:35.653055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1279/2577] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:35 2022] DELETE /identity/v3/users/c0b6f4cfede44b2aa8f4a8dfc1e211d6 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:35.685358 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a76194-f218-4b6e-9fb0-312b544f7bba None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.736598 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.737699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.737984 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.738318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.740895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.772013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:35.808349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.809833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.820968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1299/2578] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:35 2022] GET /identity/v3/auth/tokens => generated 3295 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:35.830967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.832390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.886604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:35.926993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf4a161-80bb-4ce9-a193-5b9f52aa0e2c admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.928282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:35.929728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.935090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.936229 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.937527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.937527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] PATH_INFO: `/v3/projects/54b5133f102c4606ac7981542949b178` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.938447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=54b5133f102c4606ac7981542949b178)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:35.941001 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:35.941239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c845d6-c21d-41d1-8404-b35e77fd57fd service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:35.945990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:35.946633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:35.946916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:35.948142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:35.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:35.996656 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54b5133f102c4606ac7981542949b178', 'request_id': 'req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:36.003870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54b5133f102c4606ac7981542949b178', 'request_id': 'req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:36.014571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54b5133f102c4606ac7981542949b178', 'request_id': 'req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:36.015597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.017146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58770149-6d8b-4ca7-8ecc-53fb8fd738fb service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.028018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1300/2579] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:35 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:36.028995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] The token cache is being invalidate because project 54b5133f102c4606ac7981542949b178 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:36.029336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:36.030107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-07137d01-5b6c-4653-8dbb-b57e9fd37c4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:36.031693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1280/2580] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:35 2022] DELETE /identity/v3/projects/54b5133f102c4606ac7981542949b178 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:36.037297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.037690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.037979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.056101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:36.057331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:36.123696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1301/2581] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:36.132584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.133004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.133197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.153691 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:36.154901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:36.168138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1281/2582] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:36.172386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.182582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35458edc-d497-4792-825d-c4afb1a849ff None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.187254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.187904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.188190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.188468 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.189264 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:36.189579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.191196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.203294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1302/2583] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:12:36.207960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.217272 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f955bd03-d825-4484-b72e-eff168c8128e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.222467 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.223110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.223393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.223667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.224365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.226303 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.252276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1282/2584] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:36 2022] POST /identity/v3/projects => generated 405 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:36.256295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.266140 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dd3460-9d17-4fd7-8141-62922c7b94a4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.311662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:36.321800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.322453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.322735 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.323006 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.323689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.325611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.345090 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1303/2585] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:36 2022] POST /identity/v3/users => generated 303 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:36.349688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.359836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66025fb-c562-43d2-bf40-7b8e7b989573 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.364536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.365224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.365510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.365788 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.366510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.368123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.378601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1283/2586] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:36.382276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.392739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055cfbd0-2acd-4cd7-8f9e-6d9b1c2c1c0d admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.397298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf7c98c-8acb-477e-89e6-d9f173848607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.397937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf7c98c-8acb-477e-89e6-d9f173848607 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.398219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf7c98c-8acb-477e-89e6-d9f173848607 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.398518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf7c98c-8acb-477e-89e6-d9f173848607 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.399229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf7c98c-8acb-477e-89e6-d9f173848607 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.400831 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf7c98c-8acb-477e-89e6-d9f173848607 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.411338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1304/2587] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:36.415222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.425004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e294e08e-fc8e-4ca2-965b-00d12a903266 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.429501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.430359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.430638 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.430931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] PATH_INFO: `/v3/projects/5430778e2cc8466cab37bfd899a0dceb/users/dcae68a284ce4a5db3bc8bd4acd0cec4/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.447192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5430778e2cc8466cab37bfd899a0dceb, user_id=dcae68a284ce4a5db3bc8bd4acd0cec4, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.450115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.463391 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1284/2588] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:36 2022] PUT /identity/v3/projects/5430778e2cc8466cab37bfd899a0dceb/users/dcae68a284ce4a5db3bc8bd4acd0cec4/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:36.468340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7869dab4-7bba-439c-8685-7098b5e6b669 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.468629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7869dab4-7bba-439c-8685-7098b5e6b669 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.468942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7869dab4-7bba-439c-8685-7098b5e6b669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.486689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7869dab4-7bba-439c-8685-7098b5e6b669 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:36.487876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7869dab4-7bba-439c-8685-7098b5e6b669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:36.546054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1305/2589] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:36.555697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:36.557411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:36.613205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b4fade-32cf-4819-b2bd-5dc242340337 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:36.616833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:36.617472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:36.617752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:36.618024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:36.627365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:36.628816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:36.637996 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1285/2590] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:37.310079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.310575 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.311005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.359365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] MFA Rules not processed for user `dcae68a284ce4a5db3bc8bd4acd0cec4`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:37.360609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:37.484766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1306/2591] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:37 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:37.485584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:37.497650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d1958-2c41-414d-9a5d-135f48d54553 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:37.506645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:37.508208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.509047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.509047 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.510073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:37.512311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:37.519274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:37.559797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1286/2592] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:37 2022] POST /identity/v3/projects => generated 405 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:37.563965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:37.575291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1021685-816c-40d3-bec2-d9f966f727f5 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:37.634606 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:37.635278 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.635564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.635847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.636578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:37.638639 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:37.645467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:37.666506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1307/2593] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:37 2022] POST /identity/v3/users => generated 309 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:37.668624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:37.682590 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06dd69-44c8-481a-8d25-8e0012881cc8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:37.687628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:37.688180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.688464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.688768 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.689496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:37.691367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:37.702527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1287/2594] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:37.708021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:37.717948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7897403-613b-435b-a03a-233a59291c78 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:37.723056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:37.723933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.724216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.724487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] PATH_INFO: `/v3/projects/bb8964fa0ebe4eb6aadd280d0a25378f/users/df0e7c49ca5b4e4f9009504f6167cf38/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.740301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb8964fa0ebe4eb6aadd280d0a25378f, user_id=df0e7c49ca5b4e4f9009504f6167cf38, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:37.743022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:37.756901 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1308/2595] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:37 2022] PUT /identity/v3/projects/bb8964fa0ebe4eb6aadd280d0a25378f/users/df0e7c49ca5b4e4f9009504f6167cf38/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:37.761200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:37.772810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5ae11c-6ccf-4ff6-9791-b8f8c79e1368 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:37.832152 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b704ee-0a29-4dc0-ae72-cf99c47833e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:37.832764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b704ee-0a29-4dc0-ae72-cf99c47833e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.833054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b704ee-0a29-4dc0-ae72-cf99c47833e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.833332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b704ee-0a29-4dc0-ae72-cf99c47833e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.834056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b704ee-0a29-4dc0-ae72-cf99c47833e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:37.835642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b704ee-0a29-4dc0-ae72-cf99c47833e4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:37.846148 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1288/2596] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:37 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:37.850349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:37.860861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19239a1-535d-4cac-9df5-3e3e1f915edc admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:37.865698 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:37.866612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:37.867048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:37.867218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] PATH_INFO: `/v3/projects/bb8964fa0ebe4eb6aadd280d0a25378f/users/df0e7c49ca5b4e4f9009504f6167cf38/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:37.869337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb8964fa0ebe4eb6aadd280d0a25378f, user_id=df0e7c49ca5b4e4f9009504f6167cf38, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:37.872010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:37.883556 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1309/2597] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:37 2022] PUT /identity/v3/projects/bb8964fa0ebe4eb6aadd280d0a25378f/users/df0e7c49ca5b4e4f9009504f6167cf38/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:38.514617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e80528-70fa-40f1-a7b0-09a5b3a0c74e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:38.515119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e80528-70fa-40f1-a7b0-09a5b3a0c74e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:38.515324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e80528-70fa-40f1-a7b0-09a5b3a0c74e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:38.542945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-62e80528-70fa-40f1-a7b0-09a5b3a0c74e None None] MFA Rules not processed for user `df0e7c49ca5b4e4f9009504f6167cf38`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:38.544189 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-62e80528-70fa-40f1-a7b0-09a5b3a0c74e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:38.633978 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1289/2598] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:38 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:38.839616 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:38.841810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:38.916753 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e60d79-1ec5-47b9-9994-1492ae319eff admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:38.921422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:38.922379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:38.922980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:38.923889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] PATH_INFO: `/v3/users/007d3e3d344f4e8bbd7b5d917210da39` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:38.925353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] RBAC: Authorizing `identity:delete_user(user_id=007d3e3d344f4e8bbd7b5d917210da39)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:38.927387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:38.992804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '007d3e3d344f4e8bbd7b5d917210da39'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:39.002279 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '007d3e3d344f4e8bbd7b5d917210da39'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:39.010375 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '007d3e3d344f4e8bbd7b5d917210da39'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:39.011047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '007d3e3d344f4e8bbd7b5d917210da39'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:39.011653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '007d3e3d344f4e8bbd7b5d917210da39'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:39.033369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1310/2599] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:38 2022] DELETE /identity/v3/users/007d3e3d344f4e8bbd7b5d917210da39 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:39.036267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:39.036902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:39.038018 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:39.057279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:39.058629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:39.118865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:39.134013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1290/2600] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:39.140334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:39.141978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:39.186757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:INSERT=6,keystone:SELECT=84,keystone:DELETE=10 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:39.223868 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bf4778-799b-46bc-827a-cb01057c6163 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:39.228514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:39.229193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:39.229881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:39.229881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:39.249188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:39.250953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:39.267877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1311/2601] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:39.880547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:39.883122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:39.910460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfda797-73b5-4934-ba80-3c947bfa5e0a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:39.921053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:39.923890 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:39.924985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:39.925258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] PATH_INFO: `/v3/projects/bd8151f6db1545caabc71b804f5b2773` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:39.928351 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd8151f6db1545caabc71b804f5b2773)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:39.930941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:40.039387 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd8151f6db1545caabc71b804f5b2773', 'request_id': 'req-fa41d65d-47ce-4680-a6c0-2320bcee2a63'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.047008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd8151f6db1545caabc71b804f5b2773', 'request_id': 'req-fa41d65d-47ce-4680-a6c0-2320bcee2a63'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.059687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd8151f6db1545caabc71b804f5b2773', 'request_id': 'req-fa41d65d-47ce-4680-a6c0-2320bcee2a63'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.078321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] The token cache is being invalidate because project bd8151f6db1545caabc71b804f5b2773 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:40.078321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.078975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1291/2602] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:39 2022] DELETE /identity/v3/projects/bd8151f6db1545caabc71b804f5b2773 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:40.085379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:40.097402 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:40.098560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3389a3-a7f0-4b28-a7b5-ff94742b38f0 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:40.100439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:40.185419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:40.186297 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:40.186581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:40.186859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] PATH_INFO: `/v3/users/395e4efd1c844a479926c9242bf250a5` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:40.188097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] RBAC: Authorizing `identity:delete_user(user_id=395e4efd1c844a479926c9242bf250a5)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:40.190424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:40.230124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa41d65d-47ce-4680-a6c0-2320bcee2a63 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:40.236931 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:40.237591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:40.237896 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:40.238170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:40.240002 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:40.247347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '395e4efd1c844a479926c9242bf250a5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.253091 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:40.255084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:40.255768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '395e4efd1c844a479926c9242bf250a5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.265351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '395e4efd1c844a479926c9242bf250a5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.266041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '395e4efd1c844a479926c9242bf250a5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.266558 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1292/2603] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:40 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:40.266752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '395e4efd1c844a479926c9242bf250a5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.267972 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:40.270013 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:40.288034 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb1f8e8-f85d-4462-a31b-fa668a2ecf44 service placement] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:40.291051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1312/2604] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:40 2022] DELETE /identity/v3/users/395e4efd1c844a479926c9242bf250a5 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:40.299006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:40.299897 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:40.300297 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:40.300578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] PATH_INFO: `/v3/users/dcae68a284ce4a5db3bc8bd4acd0cec4` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:40.302100 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=dcae68a284ce4a5db3bc8bd4acd0cec4)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:40.304008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:40.377751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:40.381934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dcae68a284ce4a5db3bc8bd4acd0cec4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.393332 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dcae68a284ce4a5db3bc8bd4acd0cec4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.406773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dcae68a284ce4a5db3bc8bd4acd0cec4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.428218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dcae68a284ce4a5db3bc8bd4acd0cec4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.429593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dcae68a284ce4a5db3bc8bd4acd0cec4'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.438203 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1293/2605] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:40 2022] DELETE /identity/v3/users/dcae68a284ce4a5db3bc8bd4acd0cec4 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:40.684327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:40.686190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:40.720182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:40.722630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:40.776719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:40.801118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8390d-e2a2-46a7-822b-83c1a89f1858 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:40.848631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e821c84-43b9-4980-954b-c3e520501f3d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:40.856145 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:40.857181 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:40.857475 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:40.857974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] PATH_INFO: `/v3/projects/5430778e2cc8466cab37bfd899a0dceb` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:40.859985 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5430778e2cc8466cab37bfd899a0dceb)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:40.862653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:40.897666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:40.898593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:40.898883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:40.899151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] PATH_INFO: `/v3/projects/985b8db18d0d48c2bda656e04a9bb00a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:40.900348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] RBAC: Authorizing `identity:delete_project(project_id=985b8db18d0d48c2bda656e04a9bb00a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:40.902277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:40.921317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5430778e2cc8466cab37bfd899a0dceb', 'request_id': 'req-97825e9f-ec16-4e97-9156-aaaa553c6c45'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.923831 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5430778e2cc8466cab37bfd899a0dceb', 'request_id': 'req-97825e9f-ec16-4e97-9156-aaaa553c6c45'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.932394 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5430778e2cc8466cab37bfd899a0dceb', 'request_id': 'req-97825e9f-ec16-4e97-9156-aaaa553c6c45'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.946665 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] The token cache is being invalidate because project 5430778e2cc8466cab37bfd899a0dceb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:40.946982 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.947627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.949262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1294/2606] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:40 2022] DELETE /identity/v3/projects/5430778e2cc8466cab37bfd899a0dceb => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:40.953213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:40.954713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:40.963298 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '985b8db18d0d48c2bda656e04a9bb00a', 'request_id': 'req-fdd1aea5-6c3d-4440-8a08-01dc7871a002'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.969789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '985b8db18d0d48c2bda656e04a9bb00a', 'request_id': 'req-fdd1aea5-6c3d-4440-8a08-01dc7871a002'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.977418 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '985b8db18d0d48c2bda656e04a9bb00a', 'request_id': 'req-fdd1aea5-6c3d-4440-8a08-01dc7871a002'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.990044 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] The token cache is being invalidate because project 985b8db18d0d48c2bda656e04a9bb00a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:40.990469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.991223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fdd1aea5-6c3d-4440-8a08-01dc7871a002 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:40.993377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1313/2607] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:40 2022] DELETE /identity/v3/projects/985b8db18d0d48c2bda656e04a9bb00a => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:40.997751 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c5ed2d-df8d-43bf-8eeb-42e7c1b3538a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:40.998047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c5ed2d-df8d-43bf-8eeb-42e7c1b3538a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:40.998337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c5ed2d-df8d-43bf-8eeb-42e7c1b3538a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.012677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:41.018420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-14c5ed2d-df8d-43bf-8eeb-42e7c1b3538a None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:41.019648 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-14c5ed2d-df8d-43bf-8eeb-42e7c1b3538a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.026500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97825e9f-ec16-4e97-9156-aaaa553c6c45 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.095104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1314/2608] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:41.097512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.098518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.098902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.099279 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] PATH_INFO: `/v3/users/df0e7c49ca5b4e4f9009504f6167cf38` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.100668 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df0e7c49ca5b4e4f9009504f6167cf38)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.103118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.103261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.103545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.103818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.129411 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:41.129838 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.144433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1315/2609] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:41.145792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.155267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be263b69-c810-4db0-bec7-3a51c4f1f540 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.160960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.161922 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.162374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.162649 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.163254 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df0e7c49ca5b4e4f9009504f6167cf38'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.163378 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:41.163690 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.165384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.177130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df0e7c49ca5b4e4f9009504f6167cf38'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.184425 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1316/2610] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:12:41.185367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df0e7c49ca5b4e4f9009504f6167cf38'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.189768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.198827 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df0e7c49ca5b4e4f9009504f6167cf38'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.199839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df0e7c49ca5b4e4f9009504f6167cf38'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.207306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1295/2611] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:40 2022] DELETE /identity/v3/users/df0e7c49ca5b4e4f9009504f6167cf38 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:41.207614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9677053c-e2f3-45af-8d91-c291e336e7f4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.217657 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:41.289890 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.290701 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.290935 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.291223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.294630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.332269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1317/2612] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/projects => generated 381 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:41.336218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.338167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.363324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361b7ab-1c5d-44a7-8e59-b6372aab4d05 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.450664 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.451364 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.451645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.451921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.452836 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.455370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.481226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1296/2613] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/users => generated 291 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:41.482798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.493484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124f94f7-8c4d-4dfa-94d9-03363db318f0 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.498318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.499146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.499444 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.499810 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.500861 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.503079 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.513859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1318/2614] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:41.514927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.517949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.518210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.528178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c888735-77f5-4783-9880-b35e77dc6fb9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.532432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd60efce-3253-4a95-9ea5-708dfd6eab35 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.533876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.534650 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.535043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.535499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.536435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.538232 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.538358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.539849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.540934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.541408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] PATH_INFO: `/v3/projects/bb8964fa0ebe4eb6aadd280d0a25378f` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.543043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb8964fa0ebe4eb6aadd280d0a25378f)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.545632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.549867 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:41.556260 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1319/2615] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:41.557281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.570845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097512b-677a-4a65-86f6-06b8ce4e2830 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.584439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e774c3-9be4-44a4-bc99-5c1f20167704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.585372 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e774c3-9be4-44a4-bc99-5c1f20167704 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.585662 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e774c3-9be4-44a4-bc99-5c1f20167704 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.585943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e774c3-9be4-44a4-bc99-5c1f20167704 admin admin] PATH_INFO: `/v3/projects/7b54db66665445e69b1738151bfc7dc1/users/f6c87db9cec64eebab0176c558f8f151/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.604793 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e774c3-9be4-44a4-bc99-5c1f20167704 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b54db66665445e69b1738151bfc7dc1, user_id=f6c87db9cec64eebab0176c558f8f151, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.607497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e774c3-9be4-44a4-bc99-5c1f20167704 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.608347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb8964fa0ebe4eb6aadd280d0a25378f', 'request_id': 'req-d941f574-f90c-4554-8632-a5b28fa82a48'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.616418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb8964fa0ebe4eb6aadd280d0a25378f', 'request_id': 'req-d941f574-f90c-4554-8632-a5b28fa82a48'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.624942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb8964fa0ebe4eb6aadd280d0a25378f', 'request_id': 'req-d941f574-f90c-4554-8632-a5b28fa82a48'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.626280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1320/2616] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:41 2022] PUT /identity/v3/projects/7b54db66665445e69b1738151bfc7dc1/users/f6c87db9cec64eebab0176c558f8f151/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:41.638133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0e088-b923-4d8e-85c5-a6cb3c312a71 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.638432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0e088-b923-4d8e-85c5-a6cb3c312a71 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.638715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0e088-b923-4d8e-85c5-a6cb3c312a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.643005 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] The token cache is being invalidate because project bb8964fa0ebe4eb6aadd280d0a25378f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:41.643306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.644626 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d941f574-f90c-4554-8632-a5b28fa82a48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:41.647276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1297/2617] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:41 2022] DELETE /identity/v3/projects/bb8964fa0ebe4eb6aadd280d0a25378f => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:41.653603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.653900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.654179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.667089 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-3af0e088-b923-4d8e-85c5-a6cb3c312a71 None None] MFA Rules not processed for user `f6c87db9cec64eebab0176c558f8f151`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:41.668893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3af0e088-b923-4d8e-85c5-a6cb3c312a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.672878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:41.674643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.740858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1298/2618] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:41.741071 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1321/2619] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:41.745553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.747042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.747906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.748219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.748719 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.763790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa2e8ca-5324-40fe-8252-7aa6f71b1489 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.767723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:41.768740 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.768948 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:41.769496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.769784 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.770183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.771162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.773362 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.776934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1322/2620] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:41.781345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.790994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544064a-95fb-428c-9d1e-00727a9da858 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.795986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.796787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.797176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.797524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.798963 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:41.799272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.800991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.806404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1299/2621] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/projects => generated 383 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:41.809865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.817596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1323/2622] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:12:41.819124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.829869 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7a6419-703c-43aa-8ec5-25970f7321e8 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.856619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:41.897868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6873e842-b509-4b86-b620-aa0581027a31 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.902653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.903328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.903608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.903883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.904569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.906505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.927746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.928653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:41.929193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.929193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.929803 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.930539 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.933036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.945658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1300/2623] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/users => generated 298 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:41.949151 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.959561 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf38e011-f1f5-4d22-a144-3c9cefabcbef admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:41.964612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:41.965302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:41.965589 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:41.965858 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:41.966579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:41.968147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:41.974610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1324/2624] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/projects => generated 369 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:41.975947 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.982170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1301/2625] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:41.983602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:41.995228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111af848-0766-41a7-a380-3ebba5a2cb80 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.047908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3687e1af-6cad-4d28-bde2-600c23e9629b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.053763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.054573 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.054852 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.055128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.055820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.057765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.072132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.073052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.073333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.073613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] PATH_INFO: `/v3/projects/1502bd382ce24f3bab508caeb80d19f7/users/1f6f4c7e20f2422d8605c854d742b7cf/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.082029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1325/2626] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:41 2022] POST /identity/v3/users => generated 285 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:42.083036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.089318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1502bd382ce24f3bab508caeb80d19f7, user_id=1f6f4c7e20f2422d8605c854d742b7cf, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.091942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.092593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c3ea1-b71d-4918-85a7-4c878068cfb5 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.098072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.098696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.098997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.099610 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.100523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.105960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1302/2627] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:41 2022] PUT /identity/v3/projects/1502bd382ce24f3bab508caeb80d19f7/users/1f6f4c7e20f2422d8605c854d742b7cf/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:42.108826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.110822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.120748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1326/2628] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:42.123670 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.135387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8729a1-d5a4-4afe-8353-cb87d74d6f75 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.174669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae86629-f17c-444b-bb82-3c80bc2f5e97 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.179747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.180474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.180795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.181082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.181805 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.183418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.193109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.193811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.194052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.194284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1303/2629] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:42.194396 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.195051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.196723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.198469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.208351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1327/2630] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:42.209358 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029fb88-78d3-49e8-8c4a-ece6622d3a67 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.212386 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.214130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fbfd56-2c34-4672-bfe2-93b89ae2f8d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.215004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fbfd56-2c34-4672-bfe2-93b89ae2f8d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.215367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fbfd56-2c34-4672-bfe2-93b89ae2f8d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.215651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fbfd56-2c34-4672-bfe2-93b89ae2f8d5 admin admin] PATH_INFO: `/v3/projects/1502bd382ce24f3bab508caeb80d19f7/users/1f6f4c7e20f2422d8605c854d742b7cf/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.217905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fbfd56-2c34-4672-bfe2-93b89ae2f8d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1502bd382ce24f3bab508caeb80d19f7, user_id=1f6f4c7e20f2422d8605c854d742b7cf, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.221613 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fbfd56-2c34-4672-bfe2-93b89ae2f8d5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.222586 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea9055-9b19-4e23-b0ec-a9e88a8b8979 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.227541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e846461a-60e2-4b88-bcba-24df53938c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.228483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e846461a-60e2-4b88-bcba-24df53938c83 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.228780 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e846461a-60e2-4b88-bcba-24df53938c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.229061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e846461a-60e2-4b88-bcba-24df53938c83 admin admin] PATH_INFO: `/v3/projects/91a612493806423c83920380b1bf3d22/users/8fc7d690c1764faba361abc2849fb7a7/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.232765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1304/2631] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:42 2022] PUT /identity/v3/projects/1502bd382ce24f3bab508caeb80d19f7/users/1f6f4c7e20f2422d8605c854d742b7cf/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:42.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.242983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.243263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.245631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e846461a-60e2-4b88-bcba-24df53938c83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91a612493806423c83920380b1bf3d22, user_id=8fc7d690c1764faba361abc2849fb7a7, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.248366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e846461a-60e2-4b88-bcba-24df53938c83 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.252473 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:42.263545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1328/2632] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:42 2022] PUT /identity/v3/projects/91a612493806423c83920380b1bf3d22/users/8fc7d690c1764faba361abc2849fb7a7/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:42.274743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] MFA Rules not processed for user `1f6f4c7e20f2422d8605c854d742b7cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:42.276173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:42.276542 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.276867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.277155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.305686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] MFA Rules not processed for user `8fc7d690c1764faba361abc2849fb7a7`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:42.307191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:42.325988 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:42.356934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1305/2633] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:42 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:42.367923 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.369361 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:42.378254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1329/2634] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:42 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:42.382561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.430335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4974d072-f9c5-48cc-b203-dbf77120541b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.434393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.435039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.435327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.435602 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.443715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afffdf78-8390-4678-8448-81d4ba730719 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.444968 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.446397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.448412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.449096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.449383 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.449663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.450345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.452238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.456651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1306/2635] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:42.478478 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1330/2636] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:42 2022] POST /identity/v3/projects => generated 367 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:42.482455 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.493751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fb8884-c5a5-4797-b694-1ed77f1784df service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.553608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.554544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.554830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.555133 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.555849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.557802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.561596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.581683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1307/2637] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:42 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:42.582734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.592404 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e1c76-b271-479b-a5f3-3b2417b17c74 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.597471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.598121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.598405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.598688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.599417 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.601037 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.611575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1308/2638] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:42.615510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.619717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf6e16b-b710-4d21-a895-18483c4e44f8 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.623340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.623976 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.624400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.624674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.624895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad9531d-cfd9-42db-9b1a-4fd2bd3d4bb6 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.630278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.631130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.631414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.631689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] PATH_INFO: `/v3/projects/bd95a7781ec94415801d75b59d2de213/users/796b328da8a7487999e7b781a3ca3a44/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.646388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd95a7781ec94415801d75b59d2de213, user_id=796b328da8a7487999e7b781a3ca3a44, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.649035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.650181 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:42.662536 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1309/2639] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:42 2022] PUT /identity/v3/projects/bd95a7781ec94415801d75b59d2de213/users/796b328da8a7487999e7b781a3ca3a44/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:42.666384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.685320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.686746 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.696120 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1331/2640] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:42.708526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:42.731139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62ce331-cb75-4e41-b9d7-b7772949de50 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.736206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3bbeda-4845-4790-ab94-8c3c80b40fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.736899 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3bbeda-4845-4790-ab94-8c3c80b40fe1 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.737185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3bbeda-4845-4790-ab94-8c3c80b40fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.737461 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3bbeda-4845-4790-ab94-8c3c80b40fe1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.738210 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3bbeda-4845-4790-ab94-8c3c80b40fe1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.739782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3bbeda-4845-4790-ab94-8c3c80b40fe1 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.753318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1310/2641] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:42.755016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.766957 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c686c-69c3-435d-8b07-415605cc697e service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.771561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.772436 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.772795 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.773075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] PATH_INFO: `/v3/projects/bd95a7781ec94415801d75b59d2de213/users/796b328da8a7487999e7b781a3ca3a44/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.775057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd95a7781ec94415801d75b59d2de213, user_id=796b328da8a7487999e7b781a3ca3a44, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:42.777817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:42.790036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1332/2642] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:42 2022] PUT /identity/v3/projects/bd95a7781ec94415801d75b59d2de213/users/796b328da8a7487999e7b781a3ca3a44/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:42.800765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.801061 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.801344 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:42.828200 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] MFA Rules not processed for user `796b328da8a7487999e7b781a3ca3a44`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:42.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:42.905922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1311/2643] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:42 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:42.921100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:42.921100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:42.983023 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b8a7eb-5f29-484f-a8b1-6763fdf85d2f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:42.992581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:42.993335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:42.993619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:42.993895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:43.004439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:43.005883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:43.016618 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1333/2644] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:42 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:44.374078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:44.376852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:44.388182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:44.453547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fd219f-aa54-47aa-9a92-9fa3eb14a977 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:44.459324 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:44.460273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:44.460584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:44.460914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] PATH_INFO: `/v3/users/8fc7d690c1764faba361abc2849fb7a7` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:44.462163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fc7d690c1764faba361abc2849fb7a7)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:44.464236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:44.464807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe69298-9655-41be-a5c0-66db2566dc89 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:44.469031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:44.469679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:44.469991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:44.470259 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:44.472288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:44.484604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:44.486146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817176e-5db1-4d93-97fe-49aaec0e4fcb service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:44.498059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1312/2645] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:44.513954 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fc7d690c1764faba361abc2849fb7a7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:44.521776 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fc7d690c1764faba361abc2849fb7a7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:44.523873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:44.530546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fc7d690c1764faba361abc2849fb7a7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:44.531920 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fc7d690c1764faba361abc2849fb7a7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:44.532878 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fc7d690c1764faba361abc2849fb7a7'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:44.554455 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1334/2646] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:44 2022] DELETE /identity/v3/users/8fc7d690c1764faba361abc2849fb7a7 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:44.556261 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:44.556655 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:44.557057 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:44.575531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:44.576939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:44.621718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=100 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:44.644209 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1313/2647] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:44.653743 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:44.655146 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:44.713693 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a977ac-844c-4a44-9849-cdd9a215ceb4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:44.717374 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:44.718020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:44.718304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:44.718581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:44.727750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:44.729207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:44.739259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1335/2648] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:45.020769 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:45.022309 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.047092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1957ce89-ae17-4554-b85a-e18919bafe96 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:45.054081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:45.055050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.055373 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.055611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] PATH_INFO: `/v3/projects/91a612493806423c83920380b1bf3d22` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.057258 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91a612493806423c83920380b1bf3d22)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:45.059124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:45.106425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91a612493806423c83920380b1bf3d22', 'request_id': 'req-7e185dc0-2373-4139-a2ab-0eb6d89195c0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.112713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91a612493806423c83920380b1bf3d22', 'request_id': 'req-7e185dc0-2373-4139-a2ab-0eb6d89195c0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.120842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91a612493806423c83920380b1bf3d22', 'request_id': 'req-7e185dc0-2373-4139-a2ab-0eb6d89195c0'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.133122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] The token cache is being invalidate because project 91a612493806423c83920380b1bf3d22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:45.133122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.133608 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.135187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1314/2649] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:45 2022] DELETE /identity/v3/projects/91a612493806423c83920380b1bf3d22 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:45.139245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:45.149104 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a60a4c1-0e7f-4f44-9c53-3e2d301ac3d0 service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:45.219319 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:45.220748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.221152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.221432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] PATH_INFO: `/v3/users/796b328da8a7487999e7b781a3ca3a44` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.222638 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=796b328da8a7487999e7b781a3ca3a44)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:45.224582 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:45.268082 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:45.270164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.277971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '796b328da8a7487999e7b781a3ca3a44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.287406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '796b328da8a7487999e7b781a3ca3a44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.288822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e185dc0-2373-4139-a2ab-0eb6d89195c0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:45.295578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:45.296639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.297023 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.297365 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] PATH_INFO: `/v3/users/f6c87db9cec64eebab0176c558f8f151` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.298518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '796b328da8a7487999e7b781a3ca3a44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.298735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6c87db9cec64eebab0176c558f8f151)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:45.299225 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '796b328da8a7487999e7b781a3ca3a44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.299833 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-bc8b74e5-52eb-498e-b9bb-e37e77ba68ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '796b328da8a7487999e7b781a3ca3a44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.301074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:45.315750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:45.326818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1336/2650] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:45 2022] DELETE /identity/v3/users/796b328da8a7487999e7b781a3ca3a44 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:45.365551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6c87db9cec64eebab0176c558f8f151'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.373942 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6c87db9cec64eebab0176c558f8f151'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.383322 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6c87db9cec64eebab0176c558f8f151'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.397346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6c87db9cec64eebab0176c558f8f151'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.398213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6c87db9cec64eebab0176c558f8f151'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.406764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1315/2651] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:45 2022] DELETE /identity/v3/users/f6c87db9cec64eebab0176c558f8f151 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:45.411060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.411605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.411605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.440642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:45.442464 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.515636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1337/2652] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:45.545117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:45.547064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.614538 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:45.641688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1137d1f-5d74-4e13-9ad0-05acbfbcd5db admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:45.646266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ca86ca-78ca-4cf8-9a10-deb741167b6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:45.647030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ca86ca-78ca-4cf8-9a10-deb741167b6b service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.647422 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ca86ca-78ca-4cf8-9a10-deb741167b6b service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.647796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ca86ca-78ca-4cf8-9a10-deb741167b6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.664751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ca86ca-78ca-4cf8-9a10-deb741167b6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:45.668316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ca86ca-78ca-4cf8-9a10-deb741167b6b service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:45.671785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:45.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.686810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1316/2653] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:45.686979 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdad5c2-6dc4-4988-b098-adbf2a642e15 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:45.691884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:45.693825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.693825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.693825 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] PATH_INFO: `/v3/projects/bd95a7781ec94415801d75b59d2de213` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.695191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd95a7781ec94415801d75b59d2de213)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:45.697407 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:45.759266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd95a7781ec94415801d75b59d2de213', 'request_id': 'req-ae5c41d3-8464-486a-8b11-f24fc31759f5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.766010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd95a7781ec94415801d75b59d2de213', 'request_id': 'req-ae5c41d3-8464-486a-8b11-f24fc31759f5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.774433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd95a7781ec94415801d75b59d2de213', 'request_id': 'req-ae5c41d3-8464-486a-8b11-f24fc31759f5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.790782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] The token cache is being invalidate because project bd95a7781ec94415801d75b59d2de213 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:45.791469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.792245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ae5c41d3-8464-486a-8b11-f24fc31759f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:45.794118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1338/2654] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:45 2022] DELETE /identity/v3/projects/bd95a7781ec94415801d75b59d2de213 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:45.799786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.800163 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.800367 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.818572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:45.819790 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.903755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1317/2655] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:45.913423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.913931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.914485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.936774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:45.938187 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:45.947757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1339/2656] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:45.952283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:45.963245 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bff313-9835-48dc-b928-fdc64eaf5ceb None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:45.968318 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:45.968936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:45.969222 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:45.969501 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:45.970772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:45.971085 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:45.972795 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:45.989783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1318/2657] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:12:45.998272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.007002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b35b27-6633-4f98-b867-f5765553c82d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.015857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.016579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.018819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.018950 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.019397 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.019755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.020930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.023441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.033610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b836287-dc4e-40c1-946c-72f5d25178c3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.038857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.039971 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.040324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.040677 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] PATH_INFO: `/v3/projects/7b54db66665445e69b1738151bfc7dc1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.042136 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b54db66665445e69b1738151bfc7dc1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.044531 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.057158 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1340/2658] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:45 2022] POST /identity/v3/projects => generated 399 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:46.061049 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.108375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b54db66665445e69b1738151bfc7dc1', 'request_id': 'req-d8017cd8-fe6a-468f-96dd-33cb47efea74'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.124390 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b54db66665445e69b1738151bfc7dc1', 'request_id': 'req-d8017cd8-fe6a-468f-96dd-33cb47efea74'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.135089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b54db66665445e69b1738151bfc7dc1', 'request_id': 'req-d8017cd8-fe6a-468f-96dd-33cb47efea74'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.138479 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:46.151822 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] The token cache is being invalidate because project 7b54db66665445e69b1738151bfc7dc1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:46.152156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.152775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.154658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1319/2659] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:46 2022] DELETE /identity/v3/projects/7b54db66665445e69b1738151bfc7dc1 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:46.158979 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.160382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.168311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48459bb-6f46-4923-934d-b1b4a869bf1f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.225508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:46.239226 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.240910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.241416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.241846 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.242697 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.243989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8017cd8-fe6a-468f-96dd-33cb47efea74 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.245053 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.250121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.252028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.252382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.252747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] PATH_INFO: `/v3/users/1f6f4c7e20f2422d8605c854d742b7cf` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.255894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f6f4c7e20f2422d8605c854d742b7cf)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.259324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.273634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1341/2660] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:46 2022] POST /identity/v3/users => generated 300 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:46.278791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.282244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.300427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0c052d-fcea-4cb8-8443-76b6e68fb5f4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.314161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.315238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.315980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.316270 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.317172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.319331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.330773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f6f4c7e20f2422d8605c854d742b7cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.338220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1342/2661] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:46.342105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f6f4c7e20f2422d8605c854d742b7cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.345569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.351731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f6f4c7e20f2422d8605c854d742b7cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.358454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd17581-a071-4ab9-b688-d9445a432f95 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.370250 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f6f4c7e20f2422d8605c854d742b7cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.370991 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f6f4c7e20f2422d8605c854d742b7cf'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.373665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c49b5f-e51e-480f-9edf-2d2fba30dcce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.374461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c49b5f-e51e-480f-9edf-2d2fba30dcce admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.374965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c49b5f-e51e-480f-9edf-2d2fba30dcce admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.375749 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c49b5f-e51e-480f-9edf-2d2fba30dcce admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.376627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c49b5f-e51e-480f-9edf-2d2fba30dcce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.378283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1320/2662] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:46 2022] DELETE /identity/v3/users/1f6f4c7e20f2422d8605c854d742b7cf => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:46.378925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c49b5f-e51e-480f-9edf-2d2fba30dcce admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.395905 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1343/2663] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:46.400804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.402852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.494808 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01cf76-081e-4ede-add8-db982ef45775 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.501658 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.503071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.503468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.503850 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] PATH_INFO: `/v3/projects/796ee9fa69c34f9e9728e46407a34bec/users/3deef0a506ee45dbb70ef820834329e8/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.525937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] RBAC: Authorizing `identity:create_grant(project_id=796ee9fa69c34f9e9728e46407a34bec, user_id=3deef0a506ee45dbb70ef820834329e8, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.529927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.558725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1321/2664] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:46 2022] PUT /identity/v3/projects/796ee9fa69c34f9e9728e46407a34bec/users/3deef0a506ee45dbb70ef820834329e8/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:46.561010 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.561826 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.562252 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.592736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] MFA Rules not processed for user `3deef0a506ee45dbb70ef820834329e8`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:46.594695 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.680428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1344/2665] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:46 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:46.684714 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.748201 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:46.754022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.756142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.775470 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1431bc1f-63cc-404d-b1bb-e6789abcc824 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.783304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8edc82f-7d6e-4cd1-96ba-1f0ce9d6a6ed None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.789207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.789509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.790511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.790630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.790732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.790821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.790900 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] PATH_INFO: `/v3/projects/1502bd382ce24f3bab508caeb80d19f7` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.790977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.791398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.791827 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1502bd382ce24f3bab508caeb80d19f7)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.793782 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.793922 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.828938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1322/2666] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:46 2022] POST /identity/v3/projects => generated 401 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:46.830807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:46.851624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1502bd382ce24f3bab508caeb80d19f7', 'request_id': 'req-36ca3ed7-b037-446f-b7fe-ab64297fbc28'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.853254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:46.861580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1502bd382ce24f3bab508caeb80d19f7', 'request_id': 'req-36ca3ed7-b037-446f-b7fe-ab64297fbc28'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.870521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1502bd382ce24f3bab508caeb80d19f7', 'request_id': 'req-36ca3ed7-b037-446f-b7fe-ab64297fbc28'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.883687 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] The token cache is being invalidate because project 1502bd382ce24f3bab508caeb80d19f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:46.883912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.884579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-36ca3ed7-b037-446f-b7fe-ab64297fbc28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:46.886390 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1345/2667] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:46 2022] DELETE /identity/v3/projects/1502bd382ce24f3bab508caeb80d19f7 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:46.890927 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e870f31-fbc1-4f5a-b14c-59dc272738db None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.891219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e870f31-fbc1-4f5a-b14c-59dc272738db None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.891498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e870f31-fbc1-4f5a-b14c-59dc272738db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.901144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6866a7d5-286d-4df9-8959-37b577508f94 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:46.911968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5e870f31-fbc1-4f5a-b14c-59dc272738db None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:46.913208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5e870f31-fbc1-4f5a-b14c-59dc272738db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:46.989283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:46.990587 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:46.990878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:46.991156 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:46.991862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:46.993839 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:46.999355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1346/2668] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:47.008124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.008856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.009206 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.023710 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1323/2669] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:46 2022] POST /identity/v3/users => generated 307 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:47.028707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.030630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:47.033349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:47.034647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:47.045560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1347/2670] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:47.045907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de62d4d-541e-4ea4-b790-404991d7a49a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.049930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.051324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.052000 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.052333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.052616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.053498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.054977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.066019 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ed3eb-29be-4db9-ac5c-654896e2055f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.066234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1324/2671] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:47.070241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.073021 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.074059 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.074363 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.074785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.076302 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:47.077345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.080472 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.081090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814e99e2-0562-439b-b296-9f6a95e949b8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.090173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.092050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.092584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.093414 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] PATH_INFO: `/v3/projects/946016c066d44853ad4ddf494c8b5cd1/users/0edf7a8ba2874f91b635d5347ae6fe32/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.100385 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1348/2672] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:12:47.105820 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.111826 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=946016c066d44853ad4ddf494c8b5cd1, user_id=0edf7a8ba2874f91b635d5347ae6fe32, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.116076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.118966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdb77e9-74df-45e7-bdff-fa550b8996b4 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.126165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.127134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.127423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.127850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.128547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.130513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.131211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1325/2673] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:47 2022] PUT /identity/v3/projects/946016c066d44853ad4ddf494c8b5cd1/users/0edf7a8ba2874f91b635d5347ae6fe32/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:47.135095 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.164856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1349/2674] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/projects => generated 407 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:47.167034 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.189290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:47.218019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fc5d37-3b5a-4876-b9bc-214d9b0fdd81 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.237377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e53da-9bd8-4552-b644-75d916292a47 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.242291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.242968 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.243255 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.243534 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.244230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.246182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.271426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1350/2675] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/users => generated 304 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:47.282997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.291363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.292582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.293252 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.294053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.296772 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.299768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03dd4ee-d013-4236-8c33-f00fe419cf51 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.306921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.307875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.308584 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.309283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.309426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1326/2676] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:47.310441 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.312284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.315184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.326974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1351/2677] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:47.327298 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c324505-2a56-422f-a25e-71a74f187a82 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.331469 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.333159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b389fcc-6281-4aa0-82ea-91796f6c2205 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.334329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b389fcc-6281-4aa0-82ea-91796f6c2205 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.334781 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b389fcc-6281-4aa0-82ea-91796f6c2205 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.335116 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b389fcc-6281-4aa0-82ea-91796f6c2205 admin admin] PATH_INFO: `/v3/projects/946016c066d44853ad4ddf494c8b5cd1/users/0edf7a8ba2874f91b635d5347ae6fe32/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.338119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b389fcc-6281-4aa0-82ea-91796f6c2205 admin admin] RBAC: Authorizing `identity:create_grant(project_id=946016c066d44853ad4ddf494c8b5cd1, user_id=0edf7a8ba2874f91b635d5347ae6fe32, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.341817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b389fcc-6281-4aa0-82ea-91796f6c2205 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.342923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e66b51b-2188-42ac-81c6-700cbe04fb6a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.348514 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.349280 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.349591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.351057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.351860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.353742 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.365905 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1327/2678] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:47 2022] PUT /identity/v3/projects/946016c066d44853ad4ddf494c8b5cd1/users/0edf7a8ba2874f91b635d5347ae6fe32/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:47.368144 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.368468 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.371135 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.381029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1352/2679] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:47.385667 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.394723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:DELETE=3,keystone:SELECT=92,keystone:INSERT=5 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:47.399844 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f47c50-6e04-4ed3-97d8-e7d6fa66f0d3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.415887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] MFA Rules not processed for user `0edf7a8ba2874f91b635d5347ae6fe32`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:47.417885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:47.514641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d3c11a-0078-408a-91a9-760ecd176e9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.515789 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d3c11a-0078-408a-91a9-760ecd176e9b admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.516295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d3c11a-0078-408a-91a9-760ecd176e9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.516595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d3c11a-0078-408a-91a9-760ecd176e9b admin admin] PATH_INFO: `/v3/projects/a89b98ecbe4546bcbf2ebb9768a6c3ac/users/cbf76bfc3b064a80a17595629af90525/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.523212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1328/2680] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:47.536599 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.538498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:47.540211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d3c11a-0078-408a-91a9-760ecd176e9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a89b98ecbe4546bcbf2ebb9768a6c3ac, user_id=cbf76bfc3b064a80a17595629af90525, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.543110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d3c11a-0078-408a-91a9-760ecd176e9b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.561987 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1353/2681] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:47 2022] PUT /identity/v3/projects/a89b98ecbe4546bcbf2ebb9768a6c3ac/users/cbf76bfc3b064a80a17595629af90525/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:47.629851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:47.639741 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.640506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.641028 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.669908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20054bc3-7713-45ff-8cd6-f5c34db6875a None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.678288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] MFA Rules not processed for user `cbf76bfc3b064a80a17595629af90525`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:47.680382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:47.742885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.743596 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.743885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.744159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.779196 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1354/2682] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:47.784723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.842092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.846066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.871311 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1329/2683] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:47.916488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abecc39c-b1e2-4301-90b6-b89c6673f71d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:47.932086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:47.932818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:47.933092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:47.933370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:47.934074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:47.936269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:47.972902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1355/2684] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/projects => generated 407 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:47.975291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:47.996563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6007c0a-97a0-43f0-9b48-f85b5df940ca service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.000022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.005688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:48.081215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd5a4e-d9fd-4a45-835e-b1cfb991d635 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.084937 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.085566 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.085706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.085854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.086180 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.086278 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.086562 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.086849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.087766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.088005 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.089926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.116949 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1330/2685] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:47 2022] POST /identity/v3/users => generated 310 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:48.117751 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.119442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.132071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e51997-0ff9-49a3-989a-51caf28c84ec admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.136755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.137629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.137912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.138190 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] PATH_INFO: `/v3/users/246dc188d9a44b1cb38721e3dd8a863a` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.139400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=246dc188d9a44b1cb38721e3dd8a863a)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.141348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.152823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.154420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.164972 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1356/2686] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:47 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:48.165659 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.173147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:48.178850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99f6700-f0f4-499e-b4ed-b18eccb7eb94 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.184285 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.185724 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.186148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.186501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.187308 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.189082 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.197377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '246dc188d9a44b1cb38721e3dd8a863a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.200288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1357/2687] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:48.203931 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.205921 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '246dc188d9a44b1cb38721e3dd8a863a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.214325 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57beacdc-c12c-4909-b1a2-f5790d5dbc55 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.215064 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '246dc188d9a44b1cb38721e3dd8a863a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.219641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.221328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.221676 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.222018 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] PATH_INFO: `/v3/projects/dc1323671e5d4f94bf358958e3720305/users/15fee465715640769e8c8cbd3fe90a7b/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.232532 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '246dc188d9a44b1cb38721e3dd8a863a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.233627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '246dc188d9a44b1cb38721e3dd8a863a'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.239002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc1323671e5d4f94bf358958e3720305, user_id=15fee465715640769e8c8cbd3fe90a7b, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.241733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.243856 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1331/2688] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:48 2022] DELETE /identity/v3/users/246dc188d9a44b1cb38721e3dd8a863a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:48.259001 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1358/2689] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:48 2022] PUT /identity/v3/projects/dc1323671e5d4f94bf358958e3720305/users/15fee465715640769e8c8cbd3fe90a7b/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:48.263329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.264901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.339607 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80309c93-7b7c-4732-ab73-0bfd30612ef5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.408187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.408926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.409205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.409484 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.410257 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.412087 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.423312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1332/2690] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:48.427625 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.440367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab323aab-fe95-45e3-9850-ebed432cffca admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.446744 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6d98a1-a889-4024-9805-f020f259c60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.447755 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6d98a1-a889-4024-9805-f020f259c60e admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.448200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6d98a1-a889-4024-9805-f020f259c60e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.448686 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6d98a1-a889-4024-9805-f020f259c60e admin admin] PATH_INFO: `/v3/projects/dc1323671e5d4f94bf358958e3720305/users/15fee465715640769e8c8cbd3fe90a7b/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.450918 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6d98a1-a889-4024-9805-f020f259c60e admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc1323671e5d4f94bf358958e3720305, user_id=15fee465715640769e8c8cbd3fe90a7b, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.456933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.459782 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.468580 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6d98a1-a889-4024-9805-f020f259c60e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.477049 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6c6fe8-16ea-4de6-b653-35aff0d9581d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.482895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.483820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.484423 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.484724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] PATH_INFO: `/v3/projects/e1d59596ecae4e288733b90c577aa86b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.486019 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1d59596ecae4e288733b90c577aa86b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.488579 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.492745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1359/2691] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:48 2022] PUT /identity/v3/projects/dc1323671e5d4f94bf358958e3720305/users/15fee465715640769e8c8cbd3fe90a7b/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:48.503643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.503999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.504738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.541818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] MFA Rules not processed for user `15fee465715640769e8c8cbd3fe90a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:48.543346 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.548574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:48.558498 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1d59596ecae4e288733b90c577aa86b', 'request_id': 'req-cab69a5a-4f6a-4fda-a14d-13af4d68e649'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.566173 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1d59596ecae4e288733b90c577aa86b', 'request_id': 'req-cab69a5a-4f6a-4fda-a14d-13af4d68e649'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.574462 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1d59596ecae4e288733b90c577aa86b', 'request_id': 'req-cab69a5a-4f6a-4fda-a14d-13af4d68e649'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.589162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] The token cache is being invalidate because project e1d59596ecae4e288733b90c577aa86b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:48.589534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.590206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cab69a5a-4f6a-4fda-a14d-13af4d68e649 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:48.591806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1333/2692] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:48 2022] DELETE /identity/v3/projects/e1d59596ecae4e288733b90c577aa86b => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:48.597389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabd4341-464f-455b-890d-5d741535e890 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.597610 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabd4341-464f-455b-890d-5d741535e890 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.597894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabd4341-464f-455b-890d-5d741535e890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.612282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:48.623992 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-cabd4341-464f-455b-890d-5d741535e890 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:48.625277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cabd4341-464f-455b-890d-5d741535e890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.625926 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1360/2693] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:48 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:48.636888 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.638284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.686986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1334/2694] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:48.695611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.695901 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.696180 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.697024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04623e0-027c-40a2-85f0-e73e161dbb1f None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.700647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.701345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.701633 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.701909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.711882 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.713355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.714307 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:48.715493 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.723096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1361/2695] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:48.723763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1335/2696] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:48.727797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.737702 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3541ddb4-d087-4b01-b71d-9890f408ec0b service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.742238 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.742865 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.743156 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.743434 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.744179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:48.744486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.746115 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.760736 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1362/2697] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:12:48.766582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.773260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7325108f-75ca-413b-b0a5-ea67875aa439 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.778232 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.778965 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.779327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.779593 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.780294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.782388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.808479 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1336/2698] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:48 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:48.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.823061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a21273-241c-45b8-8841-b60e06f49fdc admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.883371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.884041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.884327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.884601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.885335 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.887263 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.895113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.912169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1363/2699] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:48 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:48.913299 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.926052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9784f2-3bf9-475e-8d3b-d6b955649770 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.933373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.934041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.934330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.934614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.935408 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.937629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.950763 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1364/2700] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:48.953878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57241a0-697a-496b-8ca0-0b5ca4ae9fb7 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.955271 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.958403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.959044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.959328 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.959603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.961377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:48.964866 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507a57d3-f488-4dd6-83da-f94915b2c4f7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:48.970213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:48.970851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:48.971137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:48.971450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:48.972165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:48.974013 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:48.983989 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1365/2701] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:12:48.988095 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:48.997330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ff646-66be-4197-b59d-eed88f995431 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.002863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.003706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.003986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.004253 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] PATH_INFO: `/v3/projects/53ec6218a83f418f8521ccc73e45e10a/users/9b3f248d8b514f30bfda64db4a9eebc6/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.019534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:49.022537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.024572 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c698a1-9123-4da4-9abd-4b44581b2fd8 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.037539 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1337/2702] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:48 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:49.143356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53ec6218a83f418f8521ccc73e45e10a, user_id=9b3f248d8b514f30bfda64db4a9eebc6, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.146016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.159721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1366/2703] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:48 2022] PUT /identity/v3/projects/53ec6218a83f418f8521ccc73e45e10a/users/9b3f248d8b514f30bfda64db4a9eebc6/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:49.171031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.171396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.171737 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.200845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] MFA Rules not processed for user `9b3f248d8b514f30bfda64db4a9eebc6`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:49.202266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:49.271300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1338/2704] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:49 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:49.276714 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.314481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:49.353808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4cb8d8-d0b3-48f9-98a7-918effe693c9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.358692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.359348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.359631 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.359902 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.360652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.362558 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.389787 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1367/2705] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:49 2022] POST /identity/v3/projects => generated 373 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:49.394758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.405552 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d598e6b-dbbc-4846-89ef-5910795dd6b2 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.463004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.463651 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.463871 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.464147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.464874 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.466830 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.486224 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1339/2706] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:49 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:49.490294 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.500370 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd119afb-8408-4dd1-a517-708a764ae17e admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.505729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.506376 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.506663 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.506938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.507654 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.509268 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.519617 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1368/2707] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:49.523653 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.533986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa5b9cd-f3e5-4f30-9387-8ffa4bcd8695 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.538669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.539582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.539876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.540269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] PATH_INFO: `/v3/projects/21023e65c77e45419c8e9cb1050bd062/users/aff16ae919704fbd9f9981862849ae75/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.555131 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21023e65c77e45419c8e9cb1050bd062, user_id=aff16ae919704fbd9f9981862849ae75, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.557932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.571043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1340/2708] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:49 2022] PUT /identity/v3/projects/21023e65c77e45419c8e9cb1050bd062/users/aff16ae919704fbd9f9981862849ae75/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:49.575110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.586439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6cea7-4de5-46a0-981f-769c4c12e72b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.643933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd197e8-1b41-480d-8459-63f86da9f5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.644597 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd197e8-1b41-480d-8459-63f86da9f5f5 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.644914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd197e8-1b41-480d-8459-63f86da9f5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.645195 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd197e8-1b41-480d-8459-63f86da9f5f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.645914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd197e8-1b41-480d-8459-63f86da9f5f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.647501 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd197e8-1b41-480d-8459-63f86da9f5f5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.657685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1369/2709] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:49.662251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.672825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef6dabf-916a-42a6-8e39-ecc047311bb4 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.677557 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.678439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.678728 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.679009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] PATH_INFO: `/v3/projects/21023e65c77e45419c8e9cb1050bd062/users/aff16ae919704fbd9f9981862849ae75/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.681022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21023e65c77e45419c8e9cb1050bd062, user_id=aff16ae919704fbd9f9981862849ae75, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.683645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.694603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1341/2710] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:49 2022] PUT /identity/v3/projects/21023e65c77e45419c8e9cb1050bd062/users/aff16ae919704fbd9f9981862849ae75/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:49.706190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.706646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.707002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.735105 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] MFA Rules not processed for user `aff16ae919704fbd9f9981862849ae75`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:49.736486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:49.807359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1370/2711] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:49 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:49.821265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.863612 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:49.880964 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76097b99-4365-4ac1-b45b-da0f71934650 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.884681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.885439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.885738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.886039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:49.895876 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:49.897352 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:49.906833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1342/2712] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:49 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:49.980842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:49.990933 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d844625-22d9-4147-88cb-e34bb7d75eb9 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:49.994576 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf9052d-deff-444f-b56f-dbe89644ce77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:49.995227 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf9052d-deff-444f-b56f-dbe89644ce77 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:49.995504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf9052d-deff-444f-b56f-dbe89644ce77 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:49.995774 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf9052d-deff-444f-b56f-dbe89644ce77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:50.014914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:50.052518 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf9052d-deff-444f-b56f-dbe89644ce77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:50.053967 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf9052d-deff-444f-b56f-dbe89644ce77 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:50.062547 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1371/2713] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:49 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:54.368196 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:54.369661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:54.433363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae64b2-121c-4b8b-8f1a-bc9332df020f service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:54.437870 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:54.439350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:54.440288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:54.440573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] PATH_INFO: `/v3/users/3deef0a506ee45dbb70ef820834329e8` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:54.441800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3deef0a506ee45dbb70ef820834329e8)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:54.443687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:54.488565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3deef0a506ee45dbb70ef820834329e8'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:54.494859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3deef0a506ee45dbb70ef820834329e8'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:54.502692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3deef0a506ee45dbb70ef820834329e8'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:54.514478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3deef0a506ee45dbb70ef820834329e8'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:54.515110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3deef0a506ee45dbb70ef820834329e8'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:54.521022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1343/2714] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:54 2022] DELETE /identity/v3/users/3deef0a506ee45dbb70ef820834329e8 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:54.525530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:54.525983 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:54.526075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:54.542923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:54.544137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:54.602234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1372/2715] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:54.612183 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:54.613592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:54.669757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcdd50b-031a-4945-9440-95d353814833 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:54.674079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:54.674744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:54.675157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:54.675439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:54.684323 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:54.685807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:54.695283 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1344/2716] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:54.979191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:54.981003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:54.993603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8e07f-43e3-429d-9a7b-5abbf6465b87 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:54.998216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:54.999070 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:54.999355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:54.999629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] PATH_INFO: `/v3/projects/796ee9fa69c34f9e9728e46407a34bec` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.001138 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=796ee9fa69c34f9e9728e46407a34bec)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:55.003029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:55.049143 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '796ee9fa69c34f9e9728e46407a34bec', 'request_id': 'req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.054823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '796ee9fa69c34f9e9728e46407a34bec', 'request_id': 'req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.062311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '796ee9fa69c34f9e9728e46407a34bec', 'request_id': 'req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.073215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] The token cache is being invalidate because project 796ee9fa69c34f9e9728e46407a34bec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:55.073504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.074800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.076486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1373/2717] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:54 2022] DELETE /identity/v3/projects/796ee9fa69c34f9e9728e46407a34bec => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:55.081150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:55.087950 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:55.097879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0508e807-e39c-4cf0-adcf-e97849faf722 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:55.160553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:55.161544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.161832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.162115 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] PATH_INFO: `/v3/users/0edf7a8ba2874f91b635d5347ae6fe32` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.163337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0edf7a8ba2874f91b635d5347ae6fe32)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:55.166090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:55.214527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0edf7a8ba2874f91b635d5347ae6fe32'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.222284 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0edf7a8ba2874f91b635d5347ae6fe32'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.229288 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0edf7a8ba2874f91b635d5347ae6fe32'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.241071 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0edf7a8ba2874f91b635d5347ae6fe32'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.241643 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-c75b5313-f30f-4f6b-9e16-a1729d462875 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0edf7a8ba2874f91b635d5347ae6fe32'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.247482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1345/2718] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:55 2022] DELETE /identity/v3/users/0edf7a8ba2874f91b635d5347ae6fe32 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:55.511127 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:55.512647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:55.530155 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e81f8c-77cd-4ca2-a226-b4b384f45f4b admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:55.577696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:55.597153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:55.598111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.598394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.598672 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] PATH_INFO: `/v3/projects/946016c066d44853ad4ddf494c8b5cd1` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.599859 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] RBAC: Authorizing `identity:delete_project(project_id=946016c066d44853ad4ddf494c8b5cd1)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:55.601792 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:55.651114 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '946016c066d44853ad4ddf494c8b5cd1', 'request_id': 'req-844adef2-b3e8-414a-919c-c8120c1d4104'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.656446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '946016c066d44853ad4ddf494c8b5cd1', 'request_id': 'req-844adef2-b3e8-414a-919c-c8120c1d4104'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.664416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '946016c066d44853ad4ddf494c8b5cd1', 'request_id': 'req-844adef2-b3e8-414a-919c-c8120c1d4104'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.675440 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] The token cache is being invalidate because project 946016c066d44853ad4ddf494c8b5cd1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:55.675728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.676350 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-844adef2-b3e8-414a-919c-c8120c1d4104 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:55.677898 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1374/2719] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:55 2022] DELETE /identity/v3/projects/946016c066d44853ad4ddf494c8b5cd1 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:55.683118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.683591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.701512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:55.702711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:55.761447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1346/2720] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:12:55.770439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.770729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.770999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.789995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:55.791338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:55.799646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1375/2721] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:55.803754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:55.814130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf25814-533d-437b-ba22-b00517309d5b None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:55.818675 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:55.819320 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.819605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.819881 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.820640 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:12:55.821008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:55.822615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:55.833980 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1347/2722] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:12:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 19 11:12:55.838017 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:55.846896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b911c-baf0-439f-bd9e-0ddc5b2974cf None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:55.851467 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:55.852097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.852380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.852653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.853369 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:55.855240 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:55.881016 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1376/2723] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:55 2022] POST /identity/v3/projects => generated 365 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:55.885175 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:55.895107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a63d32b-fcb3-470e-8b29-c2cd09b1f8c5 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:55.899776 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:55.953529 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:55.954192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:55.954476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:55.954752 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:55.955446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:55.957408 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:55.980855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1348/2724] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:55 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:12:55.982401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:55.996778 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22531dff-a3cf-427d-9c96-5d19c85d063a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:55.999498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.000175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.000718 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.001636 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.002353 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.003908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.014210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1377/2725] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:56.017993 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.028050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd621140-88ac-43be-9d57-a29acb6c2c03 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.032799 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bd08b5-bc60-4b64-89f2-a9d32497225c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.033457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bd08b5-bc60-4b64-89f2-a9d32497225c admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.033739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bd08b5-bc60-4b64-89f2-a9d32497225c admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.034017 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bd08b5-bc60-4b64-89f2-a9d32497225c admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.034742 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bd08b5-bc60-4b64-89f2-a9d32497225c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.036343 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bd08b5-bc60-4b64-89f2-a9d32497225c admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.046159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1349/2726] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:56.050458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.060043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572567b5-5425-42aa-b19a-57a3b9af1113 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.065380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.066236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.066522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.066797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] PATH_INFO: `/v3/projects/e7a46588dc0b41b69a495d238d04a3e0/users/979ed8c6983640699f10dca427bbeda6/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.080772 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7a46588dc0b41b69a495d238d04a3e0, user_id=979ed8c6983640699f10dca427bbeda6, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.083377 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.095438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1378/2727] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:56 2022] PUT /identity/v3/projects/e7a46588dc0b41b69a495d238d04a3e0/users/979ed8c6983640699f10dca427bbeda6/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:56.105758 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.106054 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.106335 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.133941 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] MFA Rules not processed for user `979ed8c6983640699f10dca427bbeda6`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:56.135174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:56.197766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1350/2728] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:56 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:56.202219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.272564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0cb8cd-b2d7-4645-ada4-b041f7c4c349 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.277291 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.277952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.278256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.278529 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.279208 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.281121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.295417 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:56.308459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1379/2729] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:12:56 2022] POST /identity/v3/projects => generated 365 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:56.312720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.323575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc33c20-9874-4c23-aec1-0df5c78ab8ef None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.380143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.380835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.381121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.381397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.382084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.384128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.409212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1351/2730] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:12:56 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:12:56.413352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.424126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa063e-24a5-4939-bd47-e7fa3db7704f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.428613 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.429282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.429561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.429830 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.430596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.432167 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.445546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1380/2731] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:56.447586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.457618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c52d0d-fcfa-4cb2-b49f-846c9d8db218 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.462249 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.463122 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.463457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.463733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] PATH_INFO: `/v3/projects/55dab8de7b28479384127ab92de43c8d/users/bfd4f8583cf8402a8b346ec4981e4e37/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.465958 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:56.481294 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=55dab8de7b28479384127ab92de43c8d, user_id=bfd4f8583cf8402a8b346ec4981e4e37, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.483936 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.496766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1352/2732] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:56 2022] PUT /identity/v3/projects/55dab8de7b28479384127ab92de43c8d/users/bfd4f8583cf8402a8b346ec4981e4e37/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:12:56.501092 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.510756 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93599aab-7345-4580-a6d0-faa341d1ff12 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.565634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59138107-ac53-4a00-b01e-17f7c6855884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.566256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59138107-ac53-4a00-b01e-17f7c6855884 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.566486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59138107-ac53-4a00-b01e-17f7c6855884 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.566761 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59138107-ac53-4a00-b01e-17f7c6855884 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.567480 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59138107-ac53-4a00-b01e-17f7c6855884 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.569112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59138107-ac53-4a00-b01e-17f7c6855884 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.578800 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1381/2733] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:12:56.583568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.593743 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a673a122-b29e-46c8-b6bb-156a32cb3fcb admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.598877 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.599747 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.600026 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.600293 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] PATH_INFO: `/v3/projects/55dab8de7b28479384127ab92de43c8d/users/bfd4f8583cf8402a8b346ec4981e4e37/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.602285 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=55dab8de7b28479384127ab92de43c8d, user_id=bfd4f8583cf8402a8b346ec4981e4e37, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.604892 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.614859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1353/2734] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:12:56 2022] PUT /identity/v3/projects/55dab8de7b28479384127ab92de43c8d/users/bfd4f8583cf8402a8b346ec4981e4e37/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:56.624883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.625172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.625696 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.651629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] MFA Rules not processed for user `bfd4f8583cf8402a8b346ec4981e4e37`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:56.652854 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:56.717660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1382/2735] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:56 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:56.729527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.731208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:56.790142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472e619a-38bf-4a7a-82bd-3c415733b3fd admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.793739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.794375 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.794657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.794928 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.803855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.805437 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.814716 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1354/2736] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:56 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:56.888801 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:56.898818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bac961-49cb-458c-b823-4772108b7c19 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:56.902502 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57992db-b4a4-45f9-8c87-f4f99a7652a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:56.903131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57992db-b4a4-45f9-8c87-f4f99a7652a3 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:56.903412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57992db-b4a4-45f9-8c87-f4f99a7652a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:56.903684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57992db-b4a4-45f9-8c87-f4f99a7652a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:56.960354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57992db-b4a4-45f9-8c87-f4f99a7652a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:56.961816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57992db-b4a4-45f9-8c87-f4f99a7652a3 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:56.970493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1383/2737] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:56 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:12:58.716075 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:58.717547 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:58.783031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdd44e0-259a-4a12-a60d-bf90b433c840 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:58.787821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:58.788685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:58.789004 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:58.789275 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] PATH_INFO: `/v3/users/cbf76bfc3b064a80a17595629af90525` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:58.790509 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cbf76bfc3b064a80a17595629af90525)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:58.792513 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:58.841886 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cbf76bfc3b064a80a17595629af90525'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:58.850392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cbf76bfc3b064a80a17595629af90525'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:58.857448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cbf76bfc3b064a80a17595629af90525'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:58.865059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:58.870703 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cbf76bfc3b064a80a17595629af90525'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:58.871463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cbf76bfc3b064a80a17595629af90525'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:58.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1355/2738] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:58 2022] DELETE /identity/v3/users/cbf76bfc3b064a80a17595629af90525 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:58.882980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:58.883461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:58.883566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:58.890111 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:12:58.901394 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:12:58.902524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:58.970699 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1384/2739] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:12:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:12:58.981168 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:58.982699 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:59.040256 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81d0e5-b760-4d96-9a7a-fdaea2e599c0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:59.043945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:59.044584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:59.044902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:59.045178 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:59.054433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:59.055866 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:59.065192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1356/2740] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:12:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:12:59.582345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:59.583856 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:12:59.604847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5ac4c-31aa-4984-b8ec-e178583fa8e4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:59.609710 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:59.611704 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:59.612172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:59.612450 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] PATH_INFO: `/v3/projects/a89b98ecbe4546bcbf2ebb9768a6c3ac` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:59.613818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a89b98ecbe4546bcbf2ebb9768a6c3ac)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:59.615614 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:59.663912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a89b98ecbe4546bcbf2ebb9768a6c3ac', 'request_id': 'req-8785b775-5c31-4b58-81a3-89d603b23280'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.670148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a89b98ecbe4546bcbf2ebb9768a6c3ac', 'request_id': 'req-8785b775-5c31-4b58-81a3-89d603b23280'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.678351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a89b98ecbe4546bcbf2ebb9768a6c3ac', 'request_id': 'req-8785b775-5c31-4b58-81a3-89d603b23280'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.691605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] The token cache is being invalidate because project a89b98ecbe4546bcbf2ebb9768a6c3ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:12:59.691843 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.692642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.695108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1385/2741] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:12:59 2022] DELETE /identity/v3/projects/a89b98ecbe4546bcbf2ebb9768a6c3ac => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:12:59.698843 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:12:59.710574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb4d336-efcf-4cb0-9f88-dc3ac650658d service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:12:59.775041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:12:59.775907 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:12:59.776341 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:12:59.776622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] PATH_INFO: `/v3/users/15fee465715640769e8c8cbd3fe90a7b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:12:59.777967 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15fee465715640769e8c8cbd3fe90a7b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:12:59.780089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:12:59.836336 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15fee465715640769e8c8cbd3fe90a7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.851154 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15fee465715640769e8c8cbd3fe90a7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.858214 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15fee465715640769e8c8cbd3fe90a7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.882027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15fee465715640769e8c8cbd3fe90a7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.882990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-4401a994-d59f-4830-8299-2452af4a96a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15fee465715640769e8c8cbd3fe90a7b'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:12:59.889840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1357/2742] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:12:59 2022] DELETE /identity/v3/users/15fee465715640769e8c8cbd3fe90a7b => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:00.192457 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.194388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:00.210925 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785b775-5c31-4b58-81a3-89d603b23280 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.289244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.290106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.290382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.290653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] PATH_INFO: `/v3/projects/dc1323671e5d4f94bf358958e3720305` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.291853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc1323671e5d4f94bf358958e3720305)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.293770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.330088 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:00.349624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc1323671e5d4f94bf358958e3720305', 'request_id': 'req-3ea86409-4d45-4bf7-9bf7-7220be69db4e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:00.356210 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc1323671e5d4f94bf358958e3720305', 'request_id': 'req-3ea86409-4d45-4bf7-9bf7-7220be69db4e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:00.364779 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc1323671e5d4f94bf358958e3720305', 'request_id': 'req-3ea86409-4d45-4bf7-9bf7-7220be69db4e'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:00.379193 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] The token cache is being invalidate because project dc1323671e5d4f94bf358958e3720305 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:00.379564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:00.381148 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3ea86409-4d45-4bf7-9bf7-7220be69db4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:00.383126 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1386/2743] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:00 2022] DELETE /identity/v3/projects/dc1323671e5d4f94bf358958e3720305 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:00.388138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.388430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.409162 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:00.410466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:00.421217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:00.477690 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1358/2744] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:00.485521 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.485924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.486258 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.506137 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:00.507328 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:00.515398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1387/2745] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:00.520094 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.529686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b8e8f8-6dd2-414c-b070-59875e5cc1a6 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.534903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.535551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.535833 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.536104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.536859 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:13:00.537164 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.538712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.551069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1359/2746] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:13:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:13:00.555747 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.565168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4463f709-2b2e-4416-b0b5-4c1cb0c87379 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.570336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.571052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.571330 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.571601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.572404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.574311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.602526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1388/2747] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:00 2022] POST /identity/v3/projects => generated 387 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:00.608820 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.619068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb786bb5-ecc9-4554-99d2-c87821bf892d admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.675271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.675938 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.676225 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.676504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.677233 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.679143 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.698059 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1360/2748] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:00 2022] POST /identity/v3/users => generated 294 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:00.702531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.713170 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7a9e97-cb0c-4398-a34d-494f15070258 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.717765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.718406 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.718685 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.718956 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.719665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.721272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.731356 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1389/2749] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:00.735174 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.745340 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a96f00-2b5f-43ef-b6ba-fb083ecfa747 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92af0267-6788-4364-81c6-2acd88e15765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.751720 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92af0267-6788-4364-81c6-2acd88e15765 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.752099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92af0267-6788-4364-81c6-2acd88e15765 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.752379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92af0267-6788-4364-81c6-2acd88e15765 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.753141 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92af0267-6788-4364-81c6-2acd88e15765 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.754719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92af0267-6788-4364-81c6-2acd88e15765 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.764567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1361/2750] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:13:00.769550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.780951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0f458f-b132-42a9-9108-3f5e6c9a7f88 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.786906 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.787863 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.788163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.788412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] PATH_INFO: `/v3/projects/36f9c2ca940446909e85a271d1d18e55/users/8f86a4727e6f4025923ed6e6175a6a05/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.803336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=36f9c2ca940446909e85a271d1d18e55, user_id=8f86a4727e6f4025923ed6e6175a6a05, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:00.806055 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:00.818999 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1390/2751] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:00 2022] PUT /identity/v3/projects/36f9c2ca940446909e85a271d1d18e55/users/8f86a4727e6f4025923ed6e6175a6a05/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:00.831194 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.831715 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.831825 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.859302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] MFA Rules not processed for user `8f86a4727e6f4025923ed6e6175a6a05`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:00.860497 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:00.925241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1362/2752] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:00.929677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:00.991889 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4569a853-721c-4d24-b41e-bae193d1a2ea admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:00.996495 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:00.997176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:00.997461 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:00.997737 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:00.998422 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.000306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.026393 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1391/2753] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:00 2022] POST /identity/v3/projects => generated 387 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:01.030405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.041078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d3a541-07ad-45fd-8296-46dc836956df None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.045717 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:01.099008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.099667 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.099953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.100238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.100952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.102878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.122247 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1363/2754] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:01 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:01.127077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.142797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a995b5-271e-4370-9aa4-339e76d063ea admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.148809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.149596 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.149952 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.150579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.151190 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.152930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.164688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1392/2755] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:01.168773 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.179994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94808b-355b-43cc-89c7-666c49ce76a8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.184580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.185496 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.185778 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.186048 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] PATH_INFO: `/v3/projects/9d185678a835481f90ebf8bd5c363574/users/01633ea71c094670b07e903b212e5d3e/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.202010 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d185678a835481f90ebf8bd5c363574, user_id=01633ea71c094670b07e903b212e5d3e, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.204670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.218802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1364/2756] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:01 2022] PUT /identity/v3/projects/9d185678a835481f90ebf8bd5c363574/users/01633ea71c094670b07e903b212e5d3e/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:01.222768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.232552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e91d34-25b8-40ed-91e5-0ff6c25f4b63 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.262783 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:01.296546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e82f41e-10b3-4614-adf8-7b0ab165649f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.297152 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e82f41e-10b3-4614-adf8-7b0ab165649f admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.297430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e82f41e-10b3-4614-adf8-7b0ab165649f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.298219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e82f41e-10b3-4614-adf8-7b0ab165649f admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.298943 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e82f41e-10b3-4614-adf8-7b0ab165649f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.300531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e82f41e-10b3-4614-adf8-7b0ab165649f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.310313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1393/2757] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:01.315192 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.326031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677669f-ea97-4c63-b10d-07e5082c27e7 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.330990 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.331984 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.332268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.332575 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] PATH_INFO: `/v3/projects/9d185678a835481f90ebf8bd5c363574/users/01633ea71c094670b07e903b212e5d3e/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.334577 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d185678a835481f90ebf8bd5c363574, user_id=01633ea71c094670b07e903b212e5d3e, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.337215 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.350733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1365/2758] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:01 2022] PUT /identity/v3/projects/9d185678a835481f90ebf8bd5c363574/users/01633ea71c094670b07e903b212e5d3e/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:01.358569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.358728 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.359015 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.388953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] MFA Rules not processed for user `01633ea71c094670b07e903b212e5d3e`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:01.390188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:01.459611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1394/2759] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:01 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:01.471617 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.473821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:01.533559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33fdfda-a6f3-47cf-b313-5cf9c09c677a admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.537140 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852d770d-51e4-4e52-bc41-fd39a3736d37 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.537765 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852d770d-51e4-4e52-bc41-fd39a3736d37 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.538045 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852d770d-51e4-4e52-bc41-fd39a3736d37 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.538319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852d770d-51e4-4e52-bc41-fd39a3736d37 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.547346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852d770d-51e4-4e52-bc41-fd39a3736d37 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.549425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852d770d-51e4-4e52-bc41-fd39a3736d37 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.558794 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1366/2760] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:01 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:01.784482 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:01.786222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:01.851311 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57824a8-ea0f-4972-9d28-641a14d03251 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:01.856506 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:01.857543 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.857853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.858188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] PATH_INFO: `/v3/users/979ed8c6983640699f10dca427bbeda6` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.859557 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] RBAC: Authorizing `identity:delete_user(user_id=979ed8c6983640699f10dca427bbeda6)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:01.861604 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:01.911312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '979ed8c6983640699f10dca427bbeda6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:01.915154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:01.919491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '979ed8c6983640699f10dca427bbeda6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:01.928593 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '979ed8c6983640699f10dca427bbeda6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:01.929342 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '979ed8c6983640699f10dca427bbeda6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:01.930074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '979ed8c6983640699f10dca427bbeda6'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:01.949512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1395/2761] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:01 2022] DELETE /identity/v3/users/979ed8c6983640699f10dca427bbeda6 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:01.953729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:01.954704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:01.954926 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:01.973733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:01.974975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.037219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1367/2762] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:02.046975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.048481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.100551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.102074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.110463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:02.110623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b808c34f-88cd-48b7-aeec-539b325ac981 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:02.114212 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea11d10a-812e-43e3-9f7c-6db1eb64bb5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:02.114853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea11d10a-812e-43e3-9f7c-6db1eb64bb5d service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.114977 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cc9b82-2a99-44d8-9151-70c6ecc59309 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:02.115142 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea11d10a-812e-43e3-9f7c-6db1eb64bb5d service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.115416 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea11d10a-812e-43e3-9f7c-6db1eb64bb5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.120464 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:02.121418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.121700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.121997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] PATH_INFO: `/v3/users/8f86a4727e6f4025923ed6e6175a6a05` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.123405 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f86a4727e6f4025923ed6e6175a6a05)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:02.124609 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea11d10a-812e-43e3-9f7c-6db1eb64bb5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:02.125377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:02.126352 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea11d10a-812e-43e3-9f7c-6db1eb64bb5d service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:02.136318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1396/2763] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:13:02.172687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f86a4727e6f4025923ed6e6175a6a05'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.180480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f86a4727e6f4025923ed6e6175a6a05'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.187545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f86a4727e6f4025923ed6e6175a6a05'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.200840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f86a4727e6f4025923ed6e6175a6a05'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.201623 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f86a4727e6f4025923ed6e6175a6a05'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.212098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1368/2764] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:02 2022] DELETE /identity/v3/users/8f86a4727e6f4025923ed6e6175a6a05 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:02.214470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.214936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.215331 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.232395 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:02.233740 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.295729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1397/2765] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:02.305584 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.307146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.365618 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3450368-9169-4949-83d2-f826afcca8a0 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:02.369244 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:02.369927 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.370268 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.370688 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.380855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:02.382286 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:02.392648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1369/2766] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:02.445162 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.448974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f56912-402b-41da-9d16-b1513a99556d None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:02.488934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:02.489934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.490368 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.490647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] PATH_INFO: `/v3/projects/e7a46588dc0b41b69a495d238d04a3e0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.491894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7a46588dc0b41b69a495d238d04a3e0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:02.493921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:02.571254 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7a46588dc0b41b69a495d238d04a3e0', 'request_id': 'req-79234224-faaa-48c8-a81b-b6c0f9887eb9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.588600 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7a46588dc0b41b69a495d238d04a3e0', 'request_id': 'req-79234224-faaa-48c8-a81b-b6c0f9887eb9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.601653 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7a46588dc0b41b69a495d238d04a3e0', 'request_id': 'req-79234224-faaa-48c8-a81b-b6c0f9887eb9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.604867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:02.618064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] The token cache is being invalidate because project e7a46588dc0b41b69a495d238d04a3e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:02.618361 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.620112 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.622307 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1398/2767] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:02 2022] DELETE /identity/v3/projects/e7a46588dc0b41b69a495d238d04a3e0 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:02.626855 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.636900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06648a3f-f51a-463f-9ac8-639afab07003 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:02.709476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.710961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.722630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:02.723488 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.723845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.725092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] PATH_INFO: `/v3/users/bfd4f8583cf8402a8b346ec4981e4e37` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.726604 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfd4f8583cf8402a8b346ec4981e4e37)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:02.729683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:02.755546 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:02.787573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfd4f8583cf8402a8b346ec4981e4e37'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.796835 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfd4f8583cf8402a8b346ec4981e4e37'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.805476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfd4f8583cf8402a8b346ec4981e4e37'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.810533 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79234224-faaa-48c8-a81b-b6c0f9887eb9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:02.815715 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:02.817168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:02.817629 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:02.817908 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] PATH_INFO: `/v3/projects/36f9c2ca940446909e85a271d1d18e55` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:02.819246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36f9c2ca940446909e85a271d1d18e55)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:02.819729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfd4f8583cf8402a8b346ec4981e4e37'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.820508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfd4f8583cf8402a8b346ec4981e4e37'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.821281 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:02.827274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1370/2768] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:02 2022] DELETE /identity/v3/users/bfd4f8583cf8402a8b346ec4981e4e37 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:02.871572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36f9c2ca940446909e85a271d1d18e55', 'request_id': 'req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.878150 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36f9c2ca940446909e85a271d1d18e55', 'request_id': 'req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.885585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36f9c2ca940446909e85a271d1d18e55', 'request_id': 'req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.898351 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] The token cache is being invalidate because project 36f9c2ca940446909e85a271d1d18e55 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:02.899183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.899845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:02.901777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1399/2769] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:02 2022] DELETE /identity/v3/projects/36f9c2ca940446909e85a271d1d18e55 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:02.906370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:02.908806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:02.990124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9d92a9-62d7-4883-a6de-51f44cea0689 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:03.056878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:03.057689 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.057974 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.058251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] PATH_INFO: `/v3/users/01633ea71c094670b07e903b212e5d3e` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.059466 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=01633ea71c094670b07e903b212e5d3e)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:03.061434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:03.109986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01633ea71c094670b07e903b212e5d3e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.112251 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:03.118441 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01633ea71c094670b07e903b212e5d3e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.122108 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:03.123525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:03.132973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01633ea71c094670b07e903b212e5d3e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.138002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d2f926-17c2-4658-848f-a1cf6dc14ff9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:03.143257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:03.144191 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.144566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.144939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] PATH_INFO: `/v3/projects/55dab8de7b28479384127ab92de43c8d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.146624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55dab8de7b28479384127ab92de43c8d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:03.147073 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01633ea71c094670b07e903b212e5d3e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.147891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01633ea71c094670b07e903b212e5d3e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.148936 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:03.158039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1371/2770] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:02 2022] DELETE /identity/v3/users/01633ea71c094670b07e903b212e5d3e => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:03.202462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55dab8de7b28479384127ab92de43c8d', 'request_id': 'req-dd08d424-fbc5-4e1e-8073-773b84d56675'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.208661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55dab8de7b28479384127ab92de43c8d', 'request_id': 'req-dd08d424-fbc5-4e1e-8073-773b84d56675'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.216660 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55dab8de7b28479384127ab92de43c8d', 'request_id': 'req-dd08d424-fbc5-4e1e-8073-773b84d56675'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.229174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] The token cache is being invalidate because project 55dab8de7b28479384127ab92de43c8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:03.229550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.230259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-dd08d424-fbc5-4e1e-8073-773b84d56675 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.231977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1400/2771] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:03 2022] DELETE /identity/v3/projects/55dab8de7b28479384127ab92de43c8d => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:03.460792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:03.462357 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:03.543605 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb96491-fd5d-4950-8998-65bec4b2d8aa admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:03.598645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:03.599432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.599719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.599999 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] PATH_INFO: `/v3/projects/9d185678a835481f90ebf8bd5c363574` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.601217 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d185678a835481f90ebf8bd5c363574)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:03.603132 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:03.652603 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d185678a835481f90ebf8bd5c363574', 'request_id': 'req-8ff50091-ba07-4587-be12-f74caa326461'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.658756 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d185678a835481f90ebf8bd5c363574', 'request_id': 'req-8ff50091-ba07-4587-be12-f74caa326461'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.665816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d185678a835481f90ebf8bd5c363574', 'request_id': 'req-8ff50091-ba07-4587-be12-f74caa326461'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.677238 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] The token cache is being invalidate because project 9d185678a835481f90ebf8bd5c363574 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:03.677526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.678184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ff50091-ba07-4587-be12-f74caa326461 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:03.679713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1372/2772] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:03 2022] DELETE /identity/v3/projects/9d185678a835481f90ebf8bd5c363574 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:03.684641 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.685160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.685277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.703550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:03.705401 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:03.709454 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:03.765985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1401/2773] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:03.774105 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.774389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.774660 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.781963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:03.798647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:03.799879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:03.807880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1373/2774] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:03.812177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:03.824995 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf03d9-75b7-4ee4-8cf4-0aff0f9089b1 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:03.830634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:03.831371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.831762 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.832109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.832939 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:13:03.833248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:03.834994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:03.848125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1402/2775] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:13:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:13:03.853735 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:03.863198 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c48c49-194d-4cb5-9fc6-79b4fabe08d3 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:03.868434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:03.869113 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.869393 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.869666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.870359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:03.872342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:03.903615 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1374/2776] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:03 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:03.905002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:03.915101 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5a27d-3072-4977-b95d-c818c1bd2d92 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:03.977953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:03.978637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:03.983615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:03.983615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:03.983615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:03.983615 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:04.008684 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1403/2777] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:03 2022] POST /identity/v3/users => generated 284 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:04.012431 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:04.024865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ba4f94-82b6-4ac1-b2e7-d8ead06df844 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:04.029904 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:04.030616 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:04.030902 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:04.031228 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:04.032708 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:04.034321 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:04.044848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1375/2778] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:04.048781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:04.058986 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e492f0e0-0e9c-4ed4-9c0d-853e07714895 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:04.063851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2ff869-6857-4e9f-aa2d-e6aa6e4e2915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:04.064504 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2ff869-6857-4e9f-aa2d-e6aa6e4e2915 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:04.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2ff869-6857-4e9f-aa2d-e6aa6e4e2915 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:04.065205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2ff869-6857-4e9f-aa2d-e6aa6e4e2915 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:04.065960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2ff869-6857-4e9f-aa2d-e6aa6e4e2915 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:04.067811 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2ff869-6857-4e9f-aa2d-e6aa6e4e2915 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:04.078234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1404/2779] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:04.082766 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:04.093022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bc0ff9-87d9-4d6b-89de-392430e6f801 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:04.098520 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:04.099425 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:04.099719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:04.099994 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] PATH_INFO: `/v3/projects/7f3e5ed0f6e84dc9a245d8b325079040/users/e9f2422e90a641b3b8c9856f5d30d63a/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:04.114463 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f3e5ed0f6e84dc9a245d8b325079040, user_id=e9f2422e90a641b3b8c9856f5d30d63a, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:04.117104 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:04.130053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1376/2780] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:04 2022] PUT /identity/v3/projects/7f3e5ed0f6e84dc9a245d8b325079040/users/e9f2422e90a641b3b8c9856f5d30d63a/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:04.140179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:04.140474 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:04.140785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:04.168460 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] MFA Rules not processed for user `e9f2422e90a641b3b8c9856f5d30d63a`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:04.169717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:04.231045 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1405/2781] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:04 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:04.242889 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:04.244556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:04.303620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b80535-085d-49fb-b78f-ffeafd783d84 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:04.307359 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f86b27721074978b12f34dd591362ce', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:04.308003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:04.308289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:04.308568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:04.317055 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:04.318472 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:04.327265 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1377/2782] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:04 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:14.228200 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=95,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:14.323234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=52,keystone:INSERT=2 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:36.837962 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:36.839456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:36.874231 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:36.875666 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:36.899133 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2121d0-00f2-43e1-8bba-419d81252e65 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:36.902818 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:36.903532 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:36.903896 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:36.904177 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:36.906389 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:36.947753 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc575e-a812-4417-b74e-e249d5e3998d service glance] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:36.952377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:36.953276 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:36.953564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:36.953842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] PATH_INFO: `/v3/users/9b3f248d8b514f30bfda64db4a9eebc6` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:36.955043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b3f248d8b514f30bfda64db4a9eebc6)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:36.957560 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:36.967340 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:36.968848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02772315-ee2b-468a-b9c8-19d6669aaad9 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:36.978635 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1406/2783] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:36 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:37.006350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b3f248d8b514f30bfda64db4a9eebc6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.013342 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b3f248d8b514f30bfda64db4a9eebc6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.021012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b3f248d8b514f30bfda64db4a9eebc6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.036234 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b3f248d8b514f30bfda64db4a9eebc6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.037205 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b3f248d8b514f30bfda64db4a9eebc6'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.046502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1378/2784] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:36 2022] DELETE /identity/v3/users/9b3f248d8b514f30bfda64db4a9eebc6 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:37.053981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:37.054419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:37.054791 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:37.079248 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:37.080980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:37.154122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1407/2785] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:37.171845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:37.173511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:37.244142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa47c10b-5a00-4ec6-a948-82c019d1d976 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:37.247815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:37.248574 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:37.248891 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:37.249167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:37.259269 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:37.260707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:37.270372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1379/2786] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:37.547813 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:37.549449 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:37.562873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dca4fc-d672-475e-b59c-d7ecec8aa8a2 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:37.567591 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:37.568452 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:37.568766 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:37.569052 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] PATH_INFO: `/v3/projects/53ec6218a83f418f8521ccc73e45e10a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:37.570265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53ec6218a83f418f8521ccc73e45e10a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:37.572296 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:37.619439 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53ec6218a83f418f8521ccc73e45e10a', 'request_id': 'req-053e6c77-47c5-426a-9bd4-e17f4eecb866'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.620630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:37.627192 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53ec6218a83f418f8521ccc73e45e10a', 'request_id': 'req-053e6c77-47c5-426a-9bd4-e17f4eecb866'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.634245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53ec6218a83f418f8521ccc73e45e10a', 'request_id': 'req-053e6c77-47c5-426a-9bd4-e17f4eecb866'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.645057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] The token cache is being invalidate because project 53ec6218a83f418f8521ccc73e45e10a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:37.645345 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.645965 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.647499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1408/2787] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:37 2022] DELETE /identity/v3/projects/53ec6218a83f418f8521ccc73e45e10a => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:37.651595 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:37.661697 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5761e1-c0c6-4b44-98da-aaab6a8332e0 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:37.680356 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:37.722350 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:37.723218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:37.723505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:37.723783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] PATH_INFO: `/v3/users/aff16ae919704fbd9f9981862849ae75` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:37.725052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aff16ae919704fbd9f9981862849ae75)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:37.726975 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:37.773038 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aff16ae919704fbd9f9981862849ae75'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.780101 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aff16ae919704fbd9f9981862849ae75'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.787712 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aff16ae919704fbd9f9981862849ae75'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.799353 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aff16ae919704fbd9f9981862849ae75'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.800450 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1c3137a9-7042-4806-a21b-6ebf05ad21d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aff16ae919704fbd9f9981862849ae75'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:37.806725 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1380/2788] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:37 2022] DELETE /identity/v3/users/aff16ae919704fbd9f9981862849ae75 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:38.078585 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.080012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:38.094305 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e6c77-47c5-426a-9bd4-e17f4eecb866 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.150942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.151786 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.152066 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.152338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] PATH_INFO: `/v3/projects/21023e65c77e45419c8e9cb1050bd062` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.153524 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21023e65c77e45419c8e9cb1050bd062)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.155400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.199757 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21023e65c77e45419c8e9cb1050bd062', 'request_id': 'req-35c39bc3-072b-42ab-9dd0-63418e45c9c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:38.205237 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21023e65c77e45419c8e9cb1050bd062', 'request_id': 'req-35c39bc3-072b-42ab-9dd0-63418e45c9c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:38.213490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21023e65c77e45419c8e9cb1050bd062', 'request_id': 'req-35c39bc3-072b-42ab-9dd0-63418e45c9c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:38.225709 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] The token cache is being invalidate because project 21023e65c77e45419c8e9cb1050bd062 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:38.225991 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:38.226630 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-35c39bc3-072b-42ab-9dd0-63418e45c9c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:38.228151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1409/2789] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:38 2022] DELETE /identity/v3/projects/21023e65c77e45419c8e9cb1050bd062 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:38.233079 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.233571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.233681 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.251995 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:38.253237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:38.314738 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1381/2790] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:38.323867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.324094 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.342419 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:38.343602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:38.351842 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1410/2791] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:38.356065 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.366076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5383bd7-ab67-4791-91cd-b32bbae75dd1 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.370591 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.371226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.371504 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.371775 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.372518 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:13:38.372852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.374430 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.386848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1382/2792] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:13:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:13:38.391141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.401259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad0b5c-a36d-4d3b-9a87-6280c94786b8 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.406024 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.406666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.406978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.407256 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.407953 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.409942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.434788 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1411/2793] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/projects => generated 377 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:38.439009 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.449312 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36156f9b-9739-4e40-96e1-3e7ac7bfb9f3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.475044 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:38.508433 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.509121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.509406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.509685 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.510401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.512386 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.531124 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1383/2794] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:38.535211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.545231 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa2d7f2-c8a8-4b30-a3ee-9c3332425e15 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.550277 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.550923 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.551207 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.551484 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.552218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.553829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.563894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1412/2795] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:38.568159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.578771 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c58c14b-ca9d-4904-b8c2-956405b4e958 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.583448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ec0600-589b-4c76-8f8d-209ed0409167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.584138 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec0600-589b-4c76-8f8d-209ed0409167 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.584445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec0600-589b-4c76-8f8d-209ed0409167 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.584754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec0600-589b-4c76-8f8d-209ed0409167 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.585485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ec0600-589b-4c76-8f8d-209ed0409167 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.587097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ec0600-589b-4c76-8f8d-209ed0409167 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.597161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1384/2796] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:38.601163 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.610895 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae1677-84e7-45df-a166-e9e6c004f9d1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.615894 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.616768 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.617244 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.617366 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] PATH_INFO: `/v3/projects/96626cae4f7247bb90f61681b1256364/users/6702b385566f435399fc1504be58fa96/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.633003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] RBAC: Authorizing `identity:create_grant(project_id=96626cae4f7247bb90f61681b1256364, user_id=6702b385566f435399fc1504be58fa96, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.635634 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.648824 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1413/2797] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:38 2022] PUT /identity/v3/projects/96626cae4f7247bb90f61681b1256364/users/6702b385566f435399fc1504be58fa96/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:38.659031 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.659337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.659631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.686068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] MFA Rules not processed for user `6702b385566f435399fc1504be58fa96`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:38.687267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:38.749709 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1385/2798] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:13:38.754942 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.760387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:38.817970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf719d-e236-4e74-934c-53002a90c04f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.822564 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.823209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.823491 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.823764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.824438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.826347 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.850858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1414/2799] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/projects => generated 379 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:38.854817 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.866219 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4001baa8-52ab-4e58-865f-967ead093729 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.922108 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.922779 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.923068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.923348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.924056 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.926744 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.946569 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1386/2800] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:38 2022] POST /identity/v3/users => generated 290 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:38.950581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.960712 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea7e761-103f-4397-bf44-8d33eae21106 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.965239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:38.965885 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:38.966168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:38.966442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:38.967172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:38.968771 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:38.980426 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1415/2801] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:38.984187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:38.995137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0745447c-153b-4461-a7e8-d6f5946a7690 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:38.999852 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc50a32-d6f8-40c3-9fbc-fc3de5cec7a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.000512 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc50a32-d6f8-40c3-9fbc-fc3de5cec7a0 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.001103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc50a32-d6f8-40c3-9fbc-fc3de5cec7a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.001381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc50a32-d6f8-40c3-9fbc-fc3de5cec7a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.002106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc50a32-d6f8-40c3-9fbc-fc3de5cec7a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.003687 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc50a32-d6f8-40c3-9fbc-fc3de5cec7a0 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.013647 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1387/2802] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:39.018134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.028189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993cac6c-1ce6-4fee-8775-91795bef67fe admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.032853 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.033717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.033997 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.034764 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] PATH_INFO: `/v3/projects/3eee6012a9d94e8cb2c15e97c9b27eec/users/ad5a045f924143688d1449427ba19971/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.049080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3eee6012a9d94e8cb2c15e97c9b27eec, user_id=ad5a045f924143688d1449427ba19971, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.052287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.064344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1416/2803] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:39 2022] PUT /identity/v3/projects/3eee6012a9d94e8cb2c15e97c9b27eec/users/ad5a045f924143688d1449427ba19971/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:39.074440 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.074734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.075012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.101280 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] MFA Rules not processed for user `ad5a045f924143688d1449427ba19971`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:39.102459 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:39.106382 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:39.167481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1388/2804] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:39 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:39.171970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.232094 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965bbeb5-36c0-4f31-8704-ad71ffac90f3 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.236735 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.237382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.237666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.237940 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.238619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.240498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.265100 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1417/2805] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:39 2022] POST /identity/v3/projects => generated 377 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:39.269571 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.280187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc76a26-c437-4f06-bd8d-f5305d87b083 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.336080 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.336819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.337103 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.337379 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.338078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.339986 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.363420 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1389/2806] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:39 2022] POST /identity/v3/users => generated 295 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:39.364445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.375503 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4a19a0-f21a-4ba2-bcba-0a67b867a05f admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.380563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.381245 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.381526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.381797 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.382515 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.384169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.395224 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1418/2807] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:39.399302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.409854 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec101b6b-f1af-4898-9607-65ee149ce279 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.414511 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.415345 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.415628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.415903 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] PATH_INFO: `/v3/projects/ab8cfae134f940e592f576fd1c1a87e0/users/5b4f5884cae9425397a99789f661c278/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.430271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab8cfae134f940e592f576fd1c1a87e0, user_id=5b4f5884cae9425397a99789f661c278, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.432952 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.444939 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1390/2808] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:39 2022] PUT /identity/v3/projects/ab8cfae134f940e592f576fd1c1a87e0/users/5b4f5884cae9425397a99789f661c278/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:39.448777 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.459338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d494a1e-a8eb-4944-b240-e30a9a9d25ce admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.484966 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:39.516064 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ac08ea-0b40-4d71-abba-422e36e313b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.516804 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ac08ea-0b40-4d71-abba-422e36e313b8 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.517093 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ac08ea-0b40-4d71-abba-422e36e313b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.517400 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ac08ea-0b40-4d71-abba-422e36e313b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.518121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ac08ea-0b40-4d71-abba-422e36e313b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.520239 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ac08ea-0b40-4d71-abba-422e36e313b8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.529980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1419/2809] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:39 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:39.534804 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.545206 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89162f51-de3e-4619-9fd3-597cbdc15cd9 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.550179 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.551084 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.551374 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.551650 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] PATH_INFO: `/v3/projects/ab8cfae134f940e592f576fd1c1a87e0/users/5b4f5884cae9425397a99789f661c278/roles/90dd7541fe26472fb3843454a738e154` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.553686 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab8cfae134f940e592f576fd1c1a87e0, user_id=5b4f5884cae9425397a99789f661c278, role_id=90dd7541fe26472fb3843454a738e154)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.556337 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.566704 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1391/2810] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:39 2022] PUT /identity/v3/projects/ab8cfae134f940e592f576fd1c1a87e0/users/5b4f5884cae9425397a99789f661c278/roles/90dd7541fe26472fb3843454a738e154 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:39.577769 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.578061 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.578338 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.604883 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] MFA Rules not processed for user `5b4f5884cae9425397a99789f661c278`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:39.606074 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:39.672530 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1420/2811] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:39 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:39.683505 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.685248 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:39.741582 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f0635-fdfc-4f78-844a-9305bb7cf3e3 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.745220 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.745848 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.746130 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.746401 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.756041 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.757480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.762330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:39.767502 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1392/2812] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:39 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:39.807398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:39.808862 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:39.821002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7a2301-556e-45fd-9a31-2890039206c4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:39.824595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece08706-94e2-4076-ad35-ca3e394a8563 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:39.825261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece08706-94e2-4076-ad35-ca3e394a8563 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:39.825549 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece08706-94e2-4076-ad35-ca3e394a8563 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:39.825828 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece08706-94e2-4076-ad35-ca3e394a8563 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:39.881913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece08706-94e2-4076-ad35-ca3e394a8563 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:39.883343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece08706-94e2-4076-ad35-ca3e394a8563 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:39.892165 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1421/2813] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:39 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:43.788731 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:43.865078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64575b93-57cd-47df-9bb2-c46b2e16b3a1 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:43.869601 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:43.870553 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:43.870841 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:43.871114 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] PATH_INFO: `/v3/users/6702b385566f435399fc1504be58fa96` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:43.872302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6702b385566f435399fc1504be58fa96)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:43.873914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ad4f6-2b9a-4a41-bfcf-b231739478ca None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:43.874365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ad4f6-2b9a-4a41-bfcf-b231739478ca None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:43.874448 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:43.874546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9ad4f6-2b9a-4a41-bfcf-b231739478ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:43.891537 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-1d9ad4f6-2b9a-4a41-bfcf-b231739478ca None None] MFA Rules not processed for user `a217e983effa43f3a4f84221984189c5`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:43.892748 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-1d9ad4f6-2b9a-4a41-bfcf-b231739478ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:43.933880 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6702b385566f435399fc1504be58fa96'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:43.938366 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6702b385566f435399fc1504be58fa96'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:43.946451 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6702b385566f435399fc1504be58fa96'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:43.947128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=100 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:43.959725 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1422/2814] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:43 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 19 11:13:43.960861 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6702b385566f435399fc1504be58fa96'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:43.964121 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6702b385566f435399fc1504be58fa96'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:43.970564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1393/2815] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:43 2022] DELETE /identity/v3/users/6702b385566f435399fc1504be58fa96 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:43.976086 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:43.976430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:43.976738 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:43.995909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:43.997183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:44.064432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1423/2816] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:44.074878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:44.076515 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:44.136683 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a23e1da-7818-4cd6-8594-02693a400589 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:44.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:44.141006 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:44.141289 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:44.141564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:44.151128 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:44.152541 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:44.162062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1394/2817] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:44.431944 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:44.433619 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:44.461209 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34c7621-d1cc-4a9a-82fc-13b7c49cb294 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:44.466096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:44.467169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:44.467527 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:44.467873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] PATH_INFO: `/v3/projects/96626cae4f7247bb90f61681b1256364` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:44.469301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=96626cae4f7247bb90f61681b1256364)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:44.471528 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:44.522652 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96626cae4f7247bb90f61681b1256364', 'request_id': 'req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.528750 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96626cae4f7247bb90f61681b1256364', 'request_id': 'req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.541174 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96626cae4f7247bb90f61681b1256364', 'request_id': 'req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.553317 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] The token cache is being invalidate because project 96626cae4f7247bb90f61681b1256364 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:44.553612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.554250 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.555784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1424/2818] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:44 2022] DELETE /identity/v3/projects/96626cae4f7247bb90f61681b1256364 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:44.559840 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:44.570447 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1461a9-29ce-451f-b1a6-75d8324c4a0f service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:44.591622 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:44.634945 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:44.635823 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:44.636109 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:44.636384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] PATH_INFO: `/v3/users/ad5a045f924143688d1449427ba19971` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:44.637620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad5a045f924143688d1449427ba19971)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:44.639530 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:44.688527 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad5a045f924143688d1449427ba19971'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.697157 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad5a045f924143688d1449427ba19971'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.709456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad5a045f924143688d1449427ba19971'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.725030 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad5a045f924143688d1449427ba19971'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.725762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-7849fa5d-4b15-4c16-98fb-814b5b34825d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad5a045f924143688d1449427ba19971'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:44.732485 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1395/2819] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:44 2022] DELETE /identity/v3/users/ad5a045f924143688d1449427ba19971 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:44.866977 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:44.868851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:44.883481 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc47eb-e06e-45ea-b79a-fe5cf922bc44 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:44.957807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:44.958683 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:44.958964 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:44.959235 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] PATH_INFO: `/v3/users/e9f2422e90a641b3b8c9856f5d30d63a` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:44.960489 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9f2422e90a641b3b8c9856f5d30d63a)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:44.962414 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:44.987913 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:45.022128 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9f2422e90a641b3b8c9856f5d30d63a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.029992 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9f2422e90a641b3b8c9856f5d30d63a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.038051 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9f2422e90a641b3b8c9856f5d30d63a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.038655 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9f2422e90a641b3b8c9856f5d30d63a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.039283 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9f2422e90a641b3b8c9856f5d30d63a'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.061912 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1425/2820] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:44 2022] DELETE /identity/v3/users/e9f2422e90a641b3b8c9856f5d30d63a => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:45.069551 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:45.070043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:45.070150 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:45.088912 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:45.090255 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.098499 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:45.100359 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.175661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1396/2821] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:45.183063 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:45.184457 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.185955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63aabdee-754b-4725-9fd7-1f977537b4d2 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:45.198183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:45.199056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:45.199448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:45.199733 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] PATH_INFO: `/v3/projects/3eee6012a9d94e8cb2c15e97c9b27eec` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:45.201002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3eee6012a9d94e8cb2c15e97c9b27eec)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:45.202990 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:45.238895 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:45.255324 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bbbfea-dfbb-4729-be1b-1eb0a191e8a7 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:45.259154 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3eee6012a9d94e8cb2c15e97c9b27eec', 'request_id': 'req-8c1ded38-d37e-4ebf-8767-af10f45c39e1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.265526 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3eee6012a9d94e8cb2c15e97c9b27eec', 'request_id': 'req-8c1ded38-d37e-4ebf-8767-af10f45c39e1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.274234 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3eee6012a9d94e8cb2c15e97c9b27eec', 'request_id': 'req-8c1ded38-d37e-4ebf-8767-af10f45c39e1'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.286122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] The token cache is being invalidate because project 3eee6012a9d94e8cb2c15e97c9b27eec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:45.286784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.287438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.289273 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1426/2822] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:45 2022] DELETE /identity/v3/projects/3eee6012a9d94e8cb2c15e97c9b27eec => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:45.293348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:45.294723 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.315829 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:45.316491 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:45.316887 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:45.317112 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:45.320089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.330171 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:45.370873 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ded38-d37e-4ebf-8767-af10f45c39e1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:45.375700 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:45.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:45.377178 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:45.377606 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] PATH_INFO: `/v3/users/5b4f5884cae9425397a99789f661c278` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:45.379220 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b4f5884cae9425397a99789f661c278)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:45.381176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:45.402865 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:45.405325 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:45.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1397/2823] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:13:45.437077 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b4f5884cae9425397a99789f661c278'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.445081 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b4f5884cae9425397a99789f661c278'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.452399 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b4f5884cae9425397a99789f661c278'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.453076 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b4f5884cae9425397a99789f661c278'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.453692 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b4f5884cae9425397a99789f661c278'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.471487 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1427/2824] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:45 2022] DELETE /identity/v3/users/5b4f5884cae9425397a99789f661c278 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:45.700263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:45.704878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.732230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e5ba5d-e385-4d26-a75e-251cf48d47d3 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:45.781985 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:45.784578 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:45.869819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:45.870700 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:45.870989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:45.871267 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] PATH_INFO: `/v3/projects/7f3e5ed0f6e84dc9a245d8b325079040` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:45.878534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f3e5ed0f6e84dc9a245d8b325079040)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:45.880614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:45.897259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3577f721-962d-4c98-8fd2-feaa2a839360 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:45.902337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:45.903222 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:45.903509 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:45.903781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] PATH_INFO: `/v3/projects/ab8cfae134f940e592f576fd1c1a87e0` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:45.905313 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab8cfae134f940e592f576fd1c1a87e0)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:45.907550 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:45.963526 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f3e5ed0f6e84dc9a245d8b325079040', 'request_id': 'req-1ace6f17-35a2-4928-85b1-7c239879449d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.965399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:45.973493 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:45.974707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f3e5ed0f6e84dc9a245d8b325079040', 'request_id': 'req-1ace6f17-35a2-4928-85b1-7c239879449d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.975287 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab8cfae134f940e592f576fd1c1a87e0', 'request_id': 'req-badd37df-2b7f-4b65-9fc9-f867ecc4c380'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.984337 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab8cfae134f940e592f576fd1c1a87e0', 'request_id': 'req-badd37df-2b7f-4b65-9fc9-f867ecc4c380'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.988631 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f3e5ed0f6e84dc9a245d8b325079040', 'request_id': 'req-1ace6f17-35a2-4928-85b1-7c239879449d'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:45.997637 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab8cfae134f940e592f576fd1c1a87e0', 'request_id': 'req-badd37df-2b7f-4b65-9fc9-f867ecc4c380'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.010761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] The token cache is being invalidate because project 7f3e5ed0f6e84dc9a245d8b325079040 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:46.010761 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.013453 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-1ace6f17-35a2-4928-85b1-7c239879449d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.014934 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] The token cache is being invalidate because project ab8cfae134f940e592f576fd1c1a87e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:46.016384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1398/2825] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:45 2022] DELETE /identity/v3/projects/7f3e5ed0f6e84dc9a245d8b325079040 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:46.017488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.024525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-badd37df-2b7f-4b65-9fc9-f867ecc4c380 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.026640 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1428/2826] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:45 2022] DELETE /identity/v3/projects/ab8cfae134f940e592f576fd1c1a87e0 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:46.035119 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.035460 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.035693 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.065078 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:46.066376 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:46.137237 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1399/2827] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:46.146257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.146643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.147041 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.167587 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:46.168877 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:46.177975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1429/2828] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:46.182211 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.193241 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc7f689-aae7-46c4-a1f9-a3e15616fa3f None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.198043 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.198757 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.199042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.199316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.200060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:13:46.200370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.202051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.220421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1400/2829] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:13:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:13:46.221424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.232605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfd846d-62de-42bf-b32e-fa557dbe553e None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.239517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.240189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.240470 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.240765 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.241512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.243373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.277955 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1430/2830] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:46 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 19 11:13:46.279445 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.290123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd9b6ff-c1d7-4f52-a706-896adc665e18 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.352657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.353389 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.353706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.354046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.354815 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.357022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.379483 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1401/2831] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:46 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:46.383647 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.394046 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931e23bb-48d6-4e28-a71d-e7eb1c0951dd admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.399850 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.400525 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.401057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.401531 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.402847 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.405978 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.420627 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1431/2832] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:46 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:46.425412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.435482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f2e23c-2764-42d6-813f-fd1d76eb3398 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.440627 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4e607-85a8-4078-85c0-b31a4ac98da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.441434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e607-85a8-4078-85c0-b31a4ac98da9 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.441862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e607-85a8-4078-85c0-b31a4ac98da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.442260 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e607-85a8-4078-85c0-b31a4ac98da9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.444632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4e607-85a8-4078-85c0-b31a4ac98da9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.444632 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4e607-85a8-4078-85c0-b31a4ac98da9 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.458416 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1402/2833] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:46.462329 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.473605 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd096ab-cc98-412f-b183-f75da55c9301 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.479334 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.481020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.481483 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.481775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] PATH_INFO: `/v3/projects/5f1da822df0046fe84e6441618d720ca/users/1ffcb7247b6d4ba5bfd973e0b8ba480e/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.501784 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f1da822df0046fe84e6441618d720ca, user_id=1ffcb7247b6d4ba5bfd973e0b8ba480e, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.504666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.525603 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1432/2834] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:46 2022] PUT /identity/v3/projects/5f1da822df0046fe84e6441618d720ca/users/1ffcb7247b6d4ba5bfd973e0b8ba480e/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 19 11:13:46.539578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.540242 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.540780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.580418 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] MFA Rules not processed for user `1ffcb7247b6d4ba5bfd973e0b8ba480e`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:46.581842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:46.634739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:46.676262 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1403/2835] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:46.693817 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.695980 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:46.760881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fa3895-dc19-4138-8f56-2d298e2862e7 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.764671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2cce6-341d-4f14-a0bf-a8d3d18702ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.765438 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2cce6-341d-4f14-a0bf-a8d3d18702ba service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.765798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2cce6-341d-4f14-a0bf-a8d3d18702ba service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.766151 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2cce6-341d-4f14-a0bf-a8d3d18702ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.775916 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b2cce6-341d-4f14-a0bf-a8d3d18702ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.777216 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b2cce6-341d-4f14-a0bf-a8d3d18702ba service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.788835 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1433/2836] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:46 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:46.813123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:46.885619 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f2f9e7-14c3-4ab3-bdb6-5d1e68f72038 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:46.890301 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:46.891212 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:46.891500 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:46.891777 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] PATH_INFO: `/v3/users/1ffcb7247b6d4ba5bfd973e0b8ba480e` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:46.893134 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ffcb7247b6d4ba5bfd973e0b8ba480e)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:46.895039 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:46.952763 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ffcb7247b6d4ba5bfd973e0b8ba480e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.961707 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ffcb7247b6d4ba5bfd973e0b8ba480e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.971290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ffcb7247b6d4ba5bfd973e0b8ba480e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.990869 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ffcb7247b6d4ba5bfd973e0b8ba480e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:46.992090 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ffcb7247b6d4ba5bfd973e0b8ba480e'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:47.002452 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1404/2837] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:46 2022] DELETE /identity/v3/users/1ffcb7247b6d4ba5bfd973e0b8ba480e => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:47.007367 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.007867 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.007974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.025215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:47.026384 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:47.096213 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1434/2838] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:47.106739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:47.108400 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:47.172592 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da100eb-c1b7-4aff-b6c2-07c7d6609df8 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:47.177052 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103418b2-e691-4c14-ac07-f13ac7fb142c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:47.177711 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103418b2-e691-4c14-ac07-f13ac7fb142c service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.177997 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103418b2-e691-4c14-ac07-f13ac7fb142c service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.178277 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103418b2-e691-4c14-ac07-f13ac7fb142c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.185676 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:47.188410 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103418b2-e691-4c14-ac07-f13ac7fb142c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:47.190478 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103418b2-e691-4c14-ac07-f13ac7fb142c service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:47.201796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1405/2839] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:13:47.687961 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:47.689546 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:47.701730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9ba76c-de03-42c8-ae55-c671aea7bdc4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:47.707770 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:47.708881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.709236 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.709688 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] PATH_INFO: `/v3/projects/5f1da822df0046fe84e6441618d720ca` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.710851 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f1da822df0046fe84e6441618d720ca)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:47.712910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:47.725517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:47.771219 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f1da822df0046fe84e6441618d720ca', 'request_id': 'req-fe087c6a-da78-431f-96a1-33b96a2d44cd'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:47.776975 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f1da822df0046fe84e6441618d720ca', 'request_id': 'req-fe087c6a-da78-431f-96a1-33b96a2d44cd'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:47.785911 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f1da822df0046fe84e6441618d720ca', 'request_id': 'req-fe087c6a-da78-431f-96a1-33b96a2d44cd'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:47.797884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] The token cache is being invalidate because project 5f1da822df0046fe84e6441618d720ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:47.798172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:47.798807 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-fe087c6a-da78-431f-96a1-33b96a2d44cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:47.800451 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1435/2840] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:47 2022] DELETE /identity/v3/projects/5f1da822df0046fe84e6441618d720ca => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:47.805377 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.805659 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.805963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.824158 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:47.825411 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:47.886649 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1406/2841] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:47.894816 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.895102 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.895380 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.913932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:47.915131 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:47.937354 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1436/2842] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:47.942223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:47.951956 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9a89cd-5517-4e85-bbf5-1d91f3a25005 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:47.957147 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:47.957803 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.958086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.958363 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.959089 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:13:47.959396 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:47.961024 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:47.973456 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1407/2843] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:13:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:13:47.977909 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:47.987132 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aaf7cb-48ab-423a-b2f1-3d2b756979e4 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:47.991805 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:47.992443 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:47.992752 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:47.993029 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:47.993716 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:47.995579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:48.022398 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1437/2844] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:47 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:48.027051 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:48.037159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9216754d-d0c5-4e4f-af18-f26432a63c83 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:48.092791 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:48.093412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:48.093692 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:48.093983 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:48.094669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:48.096567 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:48.115388 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1408/2845] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:48 2022] POST /identity/v3/users => generated 287 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:48.119405 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:48.130402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38229629-1a0b-44ae-b3d3-562629037ff1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:48.134932 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:48.135574 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:48.135855 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:48.136130 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:48.136915 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:48.138490 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:48.149463 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1438/2846] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:48.153348 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:48.163230 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f1dc1-8dcc-48c1-b468-cae8ecf9e697 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:48.167906 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afd76f9-222f-46a2-a2a1-1d39ec0d5cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:48.168556 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afd76f9-222f-46a2-a2a1-1d39ec0d5cea admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:48.168878 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afd76f9-222f-46a2-a2a1-1d39ec0d5cea admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:48.169159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afd76f9-222f-46a2-a2a1-1d39ec0d5cea admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:48.169879 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afd76f9-222f-46a2-a2a1-1d39ec0d5cea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:48.171446 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afd76f9-222f-46a2-a2a1-1d39ec0d5cea admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:48.181208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1409/2847] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:48 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:48.185036 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:48.195267 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f8e2a-987d-4e97-80f7-89e78ab93107 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:48.199808 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:48.200661 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:48.200981 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:48.201261 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] PATH_INFO: `/v3/projects/22cd472225b043ae81c24ceca0d78cf7/users/93d82f8bc57047fd8f345f3fdfc13724/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:48.215306 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22cd472225b043ae81c24ceca0d78cf7, user_id=93d82f8bc57047fd8f345f3fdfc13724, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:48.217959 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:48.231541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1439/2848] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:48 2022] PUT /identity/v3/projects/22cd472225b043ae81c24ceca0d78cf7/users/93d82f8bc57047fd8f345f3fdfc13724/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:48.242439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:48.242729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:48.243008 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:48.269925 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] MFA Rules not processed for user `93d82f8bc57047fd8f345f3fdfc13724`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:48.271137 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:48.322302 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:48.341792 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1410/2849] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:48 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:48.353420 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:48.355110 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:48.414720 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e69b5-ac2b-457a-ab89-41428f9a2d62 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:48.418612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a902d2d-a256-4825-908b-81a693d6ed82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:48.419266 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a902d2d-a256-4825-908b-81a693d6ed82 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:48.419612 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a902d2d-a256-4825-908b-81a693d6ed82 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:48.419897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a902d2d-a256-4825-908b-81a693d6ed82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:48.429551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a902d2d-a256-4825-908b-81a693d6ed82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:48.431020 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a902d2d-a256-4825-908b-81a693d6ed82 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:48.441974 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1440/2850] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:48 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:51.409474 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:51.469786 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e19df82-4c88-4cd0-b067-59f94e0e8e0e None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:51.474955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:51.475816 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:51.476097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:51.476372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] PATH_INFO: `/v3/users/93d82f8bc57047fd8f345f3fdfc13724` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:51.477580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93d82f8bc57047fd8f345f3fdfc13724)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:51.479469 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:51.524421 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93d82f8bc57047fd8f345f3fdfc13724'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:51.533563 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93d82f8bc57047fd8f345f3fdfc13724'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:51.541106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93d82f8bc57047fd8f345f3fdfc13724'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:51.552670 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93d82f8bc57047fd8f345f3fdfc13724'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:51.553330 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93d82f8bc57047fd8f345f3fdfc13724'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:51.559097 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1411/2851] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:13:51 2022] DELETE /identity/v3/users/93d82f8bc57047fd8f345f3fdfc13724 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:51.563646 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:51.564118 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:51.564230 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:51.581269 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:51.582579 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:51.614249 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:51.642513 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1441/2852] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:51.652629 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:51.654068 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:51.711290 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d6419-44d3-4468-9a74-13e1bf9c6561 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:51.714934 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de19c44e-0d6e-4d8a-864d-df90e1a05f55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:51.715564 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de19c44e-0d6e-4d8a-864d-df90e1a05f55 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:51.715844 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de19c44e-0d6e-4d8a-864d-df90e1a05f55 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:51.716117 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de19c44e-0d6e-4d8a-864d-df90e1a05f55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:51.724973 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de19c44e-0d6e-4d8a-864d-df90e1a05f55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:51.726442 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de19c44e-0d6e-4d8a-864d-df90e1a05f55 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:51.735639 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1412/2853] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:13:52.044087 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.045689 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:52.059602 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5799629-96b7-469e-8643-281f3b1364dc None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.064320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.065759 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.066043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.066318 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] PATH_INFO: `/v3/projects/22cd472225b043ae81c24ceca0d78cf7` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.067508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=22cd472225b043ae81c24ceca0d78cf7)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.069462 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.116665 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22cd472225b043ae81c24ceca0d78cf7', 'request_id': 'req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:52.122197 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22cd472225b043ae81c24ceca0d78cf7', 'request_id': 'req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:52.130262 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22cd472225b043ae81c24ceca0d78cf7', 'request_id': 'req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:52.141583 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] The token cache is being invalidate because project 22cd472225b043ae81c24ceca0d78cf7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80153) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:13:52.141872 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:52.142494 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-163a306a-e30e-42dc-86cb-b6bbe8bcfb30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:13:52.144113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1442/2854] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:13:52 2022] DELETE /identity/v3/projects/22cd472225b043ae81c24ceca0d78cf7 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:52.149837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.150127 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.150403 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.167628 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:52.168883 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:52.177580 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:52.229346 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1413/2855] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:52.238745 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.239038 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.239316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.258182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:52.259388 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:52.267203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1443/2856] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:52.271395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.281074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9d5b41-52c3-456d-ba9c-d36c12589905 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.285736 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.286384 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.286674 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.286955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] PATH_INFO: `/v3/domains` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.287713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80152) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:13:52.288022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.289754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.301614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1414/2857] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:13:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:13:52.305708 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.314674 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346c91aa-93ec-48d2-a23d-fddd29a136e6 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.319921 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.320559 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.320897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.321172 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] PATH_INFO: `/v3/projects` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.321860 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.323730 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.349075 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1444/2858] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:13:52 2022] POST /identity/v3/projects => generated 375 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:52.353129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.363588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bea949-3fec-4bd1-9ae2-88fbbad78e1f admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.419908 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.420534 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.420802 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.421083 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] PATH_INFO: `/v3/users` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.421767 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.423669 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.443624 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1415/2859] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:13:52 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:13:52.447809 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.458732 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aedbcb-8786-4089-ab04-f6b40f603780 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.463316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.463970 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.464257 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.464552 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.465292 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.466914 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.477265 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1445/2860] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:52.481197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.491185 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29678618-76a2-41ac-b8e9-f07608d71501 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.495755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f460289-d49c-415f-9873-66e866b8aafb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.496412 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f460289-d49c-415f-9873-66e866b8aafb admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.496719 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f460289-d49c-415f-9873-66e866b8aafb admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.497011 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f460289-d49c-415f-9873-66e866b8aafb admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.497733 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f460289-d49c-415f-9873-66e866b8aafb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.499310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f460289-d49c-415f-9873-66e866b8aafb admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.510012 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1416/2861] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:13:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:13:52.514003 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.524349 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d928f69-4b81-4591-a8ee-9dd9e8f9cfe1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.529182 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.530080 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] REQUEST_METHOD: `PUT` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.530365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.530642 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] PATH_INFO: `/v3/projects/07b723b44a80414ab4d4050ca8fb162b/users/20a0da921ce94b20ae9e462bb0e5b7b2/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.547971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07b723b44a80414ab4d4050ca8fb162b, user_id=20a0da921ce94b20ae9e462bb0e5b7b2, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.552160 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.564523 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1446/2862] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:13:52 2022] PUT /identity/v3/projects/07b723b44a80414ab4d4050ca8fb162b/users/20a0da921ce94b20ae9e462bb0e5b7b2/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:13:52.575184 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.575476 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.575754 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.602187 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] MFA Rules not processed for user `20a0da921ce94b20ae9e462bb0e5b7b2`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:13:52.603395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:52.665793 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1417/2863] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:13:52 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:13:52.676295 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:13:52.678002 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:13:52.735365 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2564-c0db-4d30-93e0-1662c26e1a21 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:13:52.739412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:13:52.740039 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:13:52.740320 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:13:52.740595 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:13:52.748781 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:13:52.750188 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:13:52.755413 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:13:52.760134 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1447/2864] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:13:52 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:14:00.384060 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:00.385540 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:00.395969 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:00.442933 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63a63ac-1053-434c-98b4-b319e772eb64 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:00.446397 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:00.447028 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:00.447310 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:00.447581 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:00.449338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:00.527764 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:00.529271 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2020c087-eea1-49df-8cb5-8138af1b30f3 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:00.539894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1418/2865] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:14:00 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:14:02.501382 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:02.579259 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc52d730-3db8-4aea-a612-d04b6ce36426 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:02.585223 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:02.586243 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:02.586589 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:02.586899 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] PATH_INFO: `/v3/users/20a0da921ce94b20ae9e462bb0e5b7b2` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:02.588272 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] RBAC: Authorizing `identity:delete_user(user_id=20a0da921ce94b20ae9e462bb0e5b7b2)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:02.590217 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:02.645488 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20a0da921ce94b20ae9e462bb0e5b7b2'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:02.654624 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20a0da921ce94b20ae9e462bb0e5b7b2'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:02.662656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20a0da921ce94b20ae9e462bb0e5b7b2'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:02.663379 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20a0da921ce94b20ae9e462bb0e5b7b2'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:02.664037 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20a0da921ce94b20ae9e462bb0e5b7b2'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:02.684099 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1448/2866] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:14:02 2022] DELETE /identity/v3/users/20a0da921ce94b20ae9e462bb0e5b7b2 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:14:02.688559 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:02.689074 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:02.689287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:02.706545 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:14:02.707932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:02.783146 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1419/2867] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:14:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:14:02.793563 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:02.795215 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:02.874969 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae776df-8bdd-4f94-a6ce-ce1ea944625a admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:02.879402 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf249d4-1ad4-4109-be63-35bcb61fe69b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:02.880729 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf249d4-1ad4-4109-be63-35bcb61fe69b service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:02.881096 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf249d4-1ad4-4109-be63-35bcb61fe69b service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:02.881446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf249d4-1ad4-4109-be63-35bcb61fe69b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:02.893508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf249d4-1ad4-4109-be63-35bcb61fe69b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:02.894971 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf249d4-1ad4-4109-be63-35bcb61fe69b service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:02.905072 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1449/2868] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:14:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:14:03.195426 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.197022 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:03.211868 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb98691-0cb6-476d-a477-6d251ec40238 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.217657 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.218648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.219027 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.219399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] PATH_INFO: `/v3/projects/07b723b44a80414ab4d4050ca8fb162b` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.220810 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=07b723b44a80414ab4d4050ca8fb162b)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.224381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.277648 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07b723b44a80414ab4d4050ca8fb162b', 'request_id': 'req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:03.281806 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:03.284661 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07b723b44a80414ab4d4050ca8fb162b', 'request_id': 'req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:03.294338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07b723b44a80414ab4d4050ca8fb162b', 'request_id': 'req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:03.306118 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] The token cache is being invalidate because project 07b723b44a80414ab4d4050ca8fb162b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:14:03.306398 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:03.307123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-b7b45625-4fb2-436a-8797-ab4ad64d5b5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:14:03.308780 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1420/2869] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:14:03 2022] DELETE /identity/v3/projects/07b723b44a80414ab4d4050ca8fb162b => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:14:03.314073 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.314429 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.315316 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.333446 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:14:03.334671 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:03.353435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:03.399355 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1450/2870] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:14:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:14:03.407139 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.407429 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.407706 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.427849 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:14:03.429092 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:03.437170 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1421/2871] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:14:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:14:03.441620 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.451169 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a6af33-5a32-43d5-909a-8ea413153b04 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.456065 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.456721 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.457012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.457284 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] PATH_INFO: `/v3/domains` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.458011 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80153) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 19 11:14:03.458312 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.459951 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.471706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1451/2872] 10.222.0.33 () {58 vars in 1152 bytes} [Tue Jul 19 11:14:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 19 11:14:03.476062 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.485434 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0af5afe-1f67-4351-978c-8b4e1f83f646 None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.490863 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.491508 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.491797 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.492076 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] PATH_INFO: `/v3/projects` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.492798 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.494698 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.519395 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1422/2873] 10.222.0.33 () {60 vars in 1152 bytes} [Tue Jul 19 11:14:03 2022] POST /identity/v3/projects => generated 383 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:14:03.523512 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.533288 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842d7caf-6c08-46cc-b1e8-cc937c6e3071 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.587876 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.588520 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.588837 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.589113 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] PATH_INFO: `/v3/users` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.589794 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.591677 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.611321 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1452/2874] 10.222.0.33 () {60 vars in 1143 bytes} [Tue Jul 19 11:14:03 2022] POST /identity/v3/users => generated 292 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 19 11:14:03.616123 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.626862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f365b-67fe-4a07-a5e1-24fedf1d464f admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.631191 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.631842 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.632126 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.632399 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.633159 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.634729 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.645544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1423/2875] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:14:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 19 11:14:03.649545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.660424 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971071c4-9cbc-454e-83c6-f61c9f17b4e9 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.664960 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8519ae-8ee5-4fb1-9e64-710c0b4299c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.665611 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8519ae-8ee5-4fb1-9e64-710c0b4299c0 admin admin] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.665893 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8519ae-8ee5-4fb1-9e64-710c0b4299c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.666168 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8519ae-8ee5-4fb1-9e64-710c0b4299c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.666881 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8519ae-8ee5-4fb1-9e64-710c0b4299c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.668456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8519ae-8ee5-4fb1-9e64-710c0b4299c0 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.678276 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1453/2876] 10.222.0.33 () {58 vars in 1121 bytes} [Tue Jul 19 11:14:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 19 11:14:03.682223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.692106 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3648fc5-687c-4cbb-9213-fba4f84bdcb6 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.696642 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.697537 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] REQUEST_METHOD: `PUT` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.697837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.698098 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] PATH_INFO: `/v3/projects/c519c231d95d4e14b2db727ea22d3b50/users/8e228b9868654a6499bce784a631ec2d/roles/168377268ae14ad882f881d40dd39b5c` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.713372 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c519c231d95d4e14b2db727ea22d3b50, user_id=8e228b9868654a6499bce784a631ec2d, role_id=168377268ae14ad882f881d40dd39b5c)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.715989 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.729432 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1424/2877] 10.222.0.33 () {60 vars in 1482 bytes} [Tue Jul 19 11:14:03 2022] PUT /identity/v3/projects/c519c231d95d4e14b2db727ea22d3b50/users/8e228b9868654a6499bce784a631ec2d/roles/168377268ae14ad882f881d40dd39b5c => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:14:03.734485 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba1db3b-6892-4460-99ce-5d5f9b49ed2e None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.734775 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba1db3b-6892-4460-99ce-5d5f9b49ed2e None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.735047 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba1db3b-6892-4460-99ce-5d5f9b49ed2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.753109 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-eba1db3b-6892-4460-99ce-5d5f9b49ed2e None None] MFA Rules not processed for user `1eaf5dcf504f41ff99cb8d55073af0f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:14:03.754304 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-eba1db3b-6892-4460-99ce-5d5f9b49ed2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:03.812289 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1454/2878] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:14:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:14:03.822088 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:03.823807 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:03.882932 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a72345-1be1-41e0-9add-7cf03cb600eb admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:03.886494 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:03.887129 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:03.887407 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:03.887679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:03.897066 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:03.898477 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:03.907783 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1425/2879] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:14:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:14:04.821097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f70caa-99b7-4c5b-9519-c925b6b0701a None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:04.821682 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f70caa-99b7-4c5b-9519-c925b6b0701a None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:04.821798 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f70caa-99b7-4c5b-9519-c925b6b0701a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:04.847910 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-a6f70caa-99b7-4c5b-9519-c925b6b0701a None None] MFA Rules not processed for user `8e228b9868654a6499bce784a631ec2d`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:14:04.849282 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-a6f70caa-99b7-4c5b-9519-c925b6b0701a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:04.868448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:04.930428 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1455/2880] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:14:04 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:14:04.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:04.946819 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:05.003960 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45a74d-cff5-4120-8cec-04c2288c8384 service neutron] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:05.007588 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:05.008223 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:05.008510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:05.008821 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:05.017197 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:05.018679 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:05.027684 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1426/2881] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:14:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:14:06.362845 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] REQUEST_METHOD: `POST` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:06.363348 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:06.363448 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:06.381497 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.auth.core [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] MFA Rules not processed for user `8e228b9868654a6499bce784a631ec2d`. Rule list: `[]` (Enabled: `True`). {{(pid=80153) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:14:06.382679 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:06.389404 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1456/2882] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:14:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:14:15.023630 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:DELETE=2,keystone:SELECT=84,keystone:INSERT=3 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:16.375566 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=22 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:33.870177 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:33.871586 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:33.883755 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fddc028-fb23-47a1-98ff-554785999f42 service cinder] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:33.887370 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:33.888003 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:33.888292 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:33.888573 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:33.890347 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:33.978851 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:33.980287 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772fe7e-ab7e-4987-adce-a2fd8c1f7e09 service cinder] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:34.001069 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1427/2883] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:14:33 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:14:43.998300 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=46 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:14:57.110043 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:14:57.120594 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f14a9a9-9c36-4a84-8d87-3bc635481980 None None] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:14:57.124211 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '102f9668a5f948f99ac4ba6fc639c8ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:14:57.124930 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:14:57.125218 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:14:57.125496 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:14:57.127545 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:14:57.140785 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:14:57.142717 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:14:57.153623 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1457/2884] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:14:57 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:07.148706 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:15:09.704713 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] REQUEST_METHOD: `POST` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:09.705213 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:09.705316 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:09.722125 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.auth.core [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] MFA Rules not processed for user `8e228b9868654a6499bce784a631ec2d`. Rule list: `[]` (Enabled: `True`). {{(pid=80152) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 19 11:15:09.723333 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:09.730274 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1428/2885] 10.222.0.33 () {58 vars in 957 bytes} [Tue Jul 19 11:15:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 19 11:15:09.973432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:09.982829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcbcda8-8cc2-41b1-bec3-724ab738e2d9 service cinder] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:09.986938 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:09.987561 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:09.987848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:09.988125 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:09.989897 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:09.999166 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:10.001486 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:10.009343 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1458/2886] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:09 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:10.323845 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:10.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:10.395739 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6c0e87-0c2b-40ce-a49a-f4106cf4b76c None None] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:10.400818 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:10.401471 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:10.401760 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:10.402042 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:10.403857 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:10.415959 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:10.417565 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:10.427439 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1429/2887] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:12.009508 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:12.010904 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:12.021996 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b81f03-b46b-49e7-83c5-f3d81b44688f service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:12.025569 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:12.026189 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:12.026471 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:12.026739 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:12.028412 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:12.038123 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:12.039522 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:12.047442 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1459/2888] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:12 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 19 11:15:17.974813 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:17.976264 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:17.992436 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c244044-046e-4758-9e4f-1395362b289c service placement] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:17.997885 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:17.998576 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:17.998862 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:17.999142 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:18.000963 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:18.013263 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:18.014828 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:18.026734 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1430/2889] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:17 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:22.043832 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=12 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:15:28.021718 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=41 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:15:36.080656 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:36.082056 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:36.099423 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cf7d65-c476-4cb8-947b-070aac48e7ff service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:36.106121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:36.106829 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:36.107122 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:36.107432 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:36.109336 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:36.122060 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:36.123643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:36.134511 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1460/2890] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:44.505578 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:44.518625 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9428f5-c704-4d0f-8287-1170ebbdbe59 service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:44.525050 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2a1c13591bab47d7ba589e14b83f969e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:44.530894 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:44.531182 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:44.531490 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:44.534227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:44.548161 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:44.549800 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:44.562291 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1431/2891] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:44 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:46.129973 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=6 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:15:47.623246 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:47.633421 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5b0665-71cd-495b-a6ee-fef0f85f415f service nova] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:47.637048 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ea109e86f77f4660b6538e0ec4676629', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:47.637601 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:47.637884 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:47.638161 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:47.640560 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:47.653551 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:47.655012 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:47.665106 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1461/2892] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:50.518046 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:50.531167 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e52ac0-d711-4196-a59e-ac8c67e96a2c service nova] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:50.538204 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:50.539218 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] REQUEST_METHOD: `GET` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:50.540035 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:50.540481 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:50.542614 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:50.558306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:50.560406 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:50.594053 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1432/2893] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:15:50 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:15:54.050994 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:54.053179 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:54.068084 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4a32e0-402a-4d27-a328-4f91cfb9b09f service neutron] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:54.074203 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:54.075147 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:54.075492 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:54.075848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] PATH_INFO: `/v3/users/a217e983effa43f3a4f84221984189c5` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:54.078183 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a217e983effa43f3a4f84221984189c5)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:54.080229 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:54.151581 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a217e983effa43f3a4f84221984189c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.162445 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a217e983effa43f3a4f84221984189c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.172517 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a217e983effa43f3a4f84221984189c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.173205 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a217e983effa43f3a4f84221984189c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.174435 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a217e983effa43f3a4f84221984189c5'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.199823 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1462/2894] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:15:54 2022] DELETE /identity/v3/users/a217e983effa43f3a4f84221984189c5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:15:54.377306 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:54.391227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d68320a-9329-42ab-bd78-3efd18f58b75 service placement] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:54.447099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:54.447947 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:54.448227 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:54.448499 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] PATH_INFO: `/v3/projects/a7109adf88ac4391aee35c150e96e0b1` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:54.449696 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7109adf88ac4391aee35c150e96e0b1)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:54.451645 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:54.496549 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7109adf88ac4391aee35c150e96e0b1', 'request_id': 'req-e1562c79-e9fa-4b6d-acff-324e025b7c89'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.502319 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7109adf88ac4391aee35c150e96e0b1', 'request_id': 'req-e1562c79-e9fa-4b6d-acff-324e025b7c89'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.510149 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7109adf88ac4391aee35c150e96e0b1', 'request_id': 'req-e1562c79-e9fa-4b6d-acff-324e025b7c89'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.521329 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] The token cache is being invalidate because project a7109adf88ac4391aee35c150e96e0b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:15:54.521611 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.522239 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.523724 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1433/2895] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:15:54 2022] DELETE /identity/v3/projects/a7109adf88ac4391aee35c150e96e0b1 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:15:54.527666 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:54.529121 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:54.542456 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec28f787-bfd1-49a5-9d5e-81d996aca6d8 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:54.597924 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:54.598772 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:54.599054 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:54.599327 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] PATH_INFO: `/v3/users/38b720fb42284c2d9892d380d7651999` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:54.600498 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38b720fb42284c2d9892d380d7651999)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:54.602427 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:54.647175 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38b720fb42284c2d9892d380d7651999'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.655590 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38b720fb42284c2d9892d380d7651999'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.660917 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:15:54.663433 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38b720fb42284c2d9892d380d7651999'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.664459 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38b720fb42284c2d9892d380d7651999'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.665453 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38b720fb42284c2d9892d380d7651999'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.683703 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1463/2896] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:15:54 2022] DELETE /identity/v3/users/38b720fb42284c2d9892d380d7651999 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:15:54.856480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:15:54.858620 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:15:54.877086 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1562c79-e9fa-4b6d-acff-324e025b7c89 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:15:54.933811 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:15:54.934837 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:15:54.935226 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:15:54.935510 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] PATH_INFO: `/v3/projects/7734ed48187647389961b70ea9d92128` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:15:54.937317 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7734ed48187647389961b70ea9d92128)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:15:54.939208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:15:54.945663 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:15:54.986762 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7734ed48187647389961b70ea9d92128', 'request_id': 'req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.992266 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7734ed48187647389961b70ea9d92128', 'request_id': 'req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:54.999937 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7734ed48187647389961b70ea9d92128', 'request_id': 'req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:55.011099 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] The token cache is being invalidate because project 7734ed48187647389961b70ea9d92128 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:15:55.011381 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:55.011998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:15:55.013544 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1434/2897] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:15:54 2022] DELETE /identity/v3/projects/7734ed48187647389961b70ea9d92128 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:16:04.680141 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=8,keystone:DELETE=2 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:16:05.008110 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=22,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:16:07.199097 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:16:07.200572 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:16:07.218458 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130d8f1-8410-462b-bdb6-5cc628eeb710 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:16:07.275004 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:16:07.275870 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:16:07.276153 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:16:07.276431 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] PATH_INFO: `/v3/users/8e228b9868654a6499bce784a631ec2d` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:16:07.277643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e228b9868654a6499bce784a631ec2d)` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:16:07.279848 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:16:07.324387 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e228b9868654a6499bce784a631ec2d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.332033 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e228b9868654a6499bce784a631ec2d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.338858 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e228b9868654a6499bce784a631ec2d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.339476 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e228b9868654a6499bce784a631ec2d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.340069 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.notifications [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e228b9868654a6499bce784a631ec2d'} {{(pid=80153) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.356958 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1464/2898] 10.222.0.33 () {58 vars in 1223 bytes} [Tue Jul 19 11:16:07 2022] DELETE /identity/v3/users/8e228b9868654a6499bce784a631ec2d => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:16:07.524832 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Authenticating user token {{(pid=80152) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:16:07.526338 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.fernet_utils [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80152) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:16:07.540955 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0cb81-3411-44aa-8e37-afd3a9d13f23 admin admin] Validating token access rules against request {{(pid=80152) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:16:07.598107 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1eaf5dcf504f41ff99cb8d55073af0f9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a1dc1eb7c79345caaeb191fa0f9c008f', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80152) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:16:07.598900 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:16:07.599208 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:16:07.599482 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] PATH_INFO: `/v3/projects/c519c231d95d4e14b2db727ea22d3b50` {{(pid=80152) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:16:07.600662 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c519c231d95d4e14b2db727ea22d3b50)` {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:16:07.602568 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] RBAC: Authorization granted {{(pid=80152) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:16:07.647354 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c519c231d95d4e14b2db727ea22d3b50', 'request_id': 'req-a77d62bd-384e-4abc-b6d1-86ab20e21a52'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.652981 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c519c231d95d4e14b2db727ea22d3b50', 'request_id': 'req-a77d62bd-384e-4abc-b6d1-86ab20e21a52'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.660392 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c519c231d95d4e14b2db727ea22d3b50', 'request_id': 'req-a77d62bd-384e-4abc-b6d1-86ab20e21a52'} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.672480 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] The token cache is being invalidate because project c519c231d95d4e14b2db727ea22d3b50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80152) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 19 11:16:07.672796 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.673427 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG keystone.notifications [None req-a77d62bd-384e-4abc-b6d1-86ab20e21a52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80152) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 19 11:16:07.674953 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: [pid: 80152|app: 0|req: 1435/2899] 10.222.0.33 () {58 vars in 1232 bytes} [Tue Jul 19 11:16:07 2022] DELETE /identity/v3/projects/c519c231d95d4e14b2db727ea22d3b50 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 19 11:16:17.353301 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=53,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:16:17.668998 dv-cinder-master-3-iscsi devstack@keystone.service[80152]: DEBUG dbcounter [-] [80152] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80152) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 19 11:18:40.322176 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Authenticating user token {{(pid=80153) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 19 11:18:40.323607 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:18:40.391643 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f05c9c-a4a2-4b1c-8bdc-f2720c2e81b1 admin admin] Validating token access rules against request {{(pid=80153) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 19 11:18:40.449415 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8234f48f37ae4384bf6c5f81c664bf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '528567e0c4c74f9ba0ff9a463cfc0218', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80153) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 19 11:18:40.450057 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] REQUEST_METHOD: `GET` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 19 11:18:40.450344 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] SCRIPT_NAME: `/identity` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 19 11:18:40.450621 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80153) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 19 11:18:40.452373 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.fernet_utils [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80153) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 19 11:18:40.463430 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 19 11:18:40.465022 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e75692-6c9b-4388-8af7-ef8e3024e6ff service placement] RBAC: Authorization granted {{(pid=80153) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 19 11:18:40.474875 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: [pid: 80153|app: 0|req: 1465/2900] 10.222.0.33 () {60 vars in 1365 bytes} [Tue Jul 19 11:18:40 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 19 11:18:50.470541 dv-cinder-master-3-iscsi devstack@keystone.service[80153]: DEBUG dbcounter [-] [80153] Writing DB stats keystone:SELECT=58 {{(pid=80153) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}